Description
When exporting form data from the synkronus portal (parquet, and anything derived from it, like a Stata .dta conversion), every field defined as number or integer in a form's JSON schema comes out completely blank/NA for every row. Text/string fields on the same forms export fine with real values.
This isn't tied to one form or app. It's in synkronus's export pipeline itself, so it affects any application built on Formulus/Formplayer that syncs through synkronus and has numeric fields in its schema. We checked a few unrelated apps and all of them have forms with number/integer fields (counts, measurements, ages, etc) that would hit the same code path and lose data on export.
The underlying data is fine. The observations.data JSONB column has real numeric values stored (checked with the same schema-inference query synkronus runs internally to build the export). So it's not a capture problem anywhere upstream, the data is written correctly on submission and only gets lost during export.
Where the problem is
In synkronus/pkg/dataexport/:
postgres.go:151, for any column classified as numeric, the export query casts it with (data ->> 'key')::numeric.lib/pq's driver (encode.go, textDecode) has decode cases for int4/int8 to int64, float4/float8 to float64, text to string, bool to bool, but no case for Postgres numeric. Values of that type fall through to the default branch and come back as raw []byte, not float64.service.go:318-326, when building the parquet column:
case "numeric":
if fb, ok := fieldBuilder.(*array.Float64Builder); ok {
if v, ok := value.(float64); ok {
fb.Append(v)
} else {
fb.AppendNull()
}
}Since the scanned value is actually []byte, the value.(float64) assertion fails and it silently appends a null. No error, no log, export reports success. That's the blank/NA cell showing up downstream.
Text columns don't hit this because they're cast with ::text, and T_text/T_varchar are handled by the driver, so value.(string) works. That's why strings export fine and numbers don't.
Suggested fix
Change the cast in postgres.go:151 from ::numeric to ::double precision, which is an OID the driver decodes natively as float64.
Since it's a shared export path, worth testing against more than one form/app. Pick a form with a mix of populated and genuinely empty numeric fields, confirm real values come through and true nulls still export as null (not 0 or an empty string). Because the failure is silent (export "succeeds", no error surfaced), it's probably worth checking how much numeric data has already been lost in past exports across projects if any have exorted yet and whether those need to be re-exported once this is fixed.
Description
When exporting form data from the synkronus portal (parquet, and anything derived from it, like a Stata .dta conversion), every field defined as
numberorintegerin a form's JSON schema comes out completely blank/NA for every row. Text/string fields on the same forms export fine with real values.This isn't tied to one form or app. It's in synkronus's export pipeline itself, so it affects any application built on Formulus/Formplayer that syncs through synkronus and has numeric fields in its schema. We checked a few unrelated apps and all of them have forms with number/integer fields (counts, measurements, ages, etc) that would hit the same code path and lose data on export.
The underlying data is fine. The
observations.dataJSONB column has real numeric values stored (checked with the same schema-inference query synkronus runs internally to build the export). So it's not a capture problem anywhere upstream, the data is written correctly on submission and only gets lost during export.Where the problem is
In
synkronus/pkg/dataexport/:postgres.go:151, for any column classified as numeric, the export query casts it with(data ->> 'key')::numeric.lib/pq's driver (encode.go,textDecode) has decode cases forint4/int8toint64,float4/float8tofloat64,texttostring,booltobool, but no case for Postgresnumeric. Values of that type fall through to the default branch and come back as raw[]byte, notfloat64.service.go:318-326, when building the parquet column:Since the scanned value is actually
[]byte, thevalue.(float64)assertion fails and it silently appends a null. No error, no log, export reports success. That's the blank/NA cell showing up downstream.Text columns don't hit this because they're cast with
::text, andT_text/T_varcharare handled by the driver, sovalue.(string)works. That's why strings export fine and numbers don't.Suggested fix
Change the cast in
postgres.go:151from::numericto::double precision, which is an OID the driver decodes natively asfloat64.Since it's a shared export path, worth testing against more than one form/app. Pick a form with a mix of populated and genuinely empty numeric fields, confirm real values come through and true nulls still export as null (not 0 or an empty string). Because the failure is silent (export "succeeds", no error surfaced), it's probably worth checking how much numeric data has already been lost in past exports across projects if any have exorted yet and whether those need to be re-exported once this is fixed.