Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 66 additions & 2 deletions documentation/components/adapters/json.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ simplified and efficient task, perfectly aligning with the robust and adaptable

## Installation

For detailed installation instructions, see the [installation page](/documentation/installation/packages/etl-adapter-json.md).
For detailed installation instructions, see
the [installation page](/documentation/installation/packages/etl-adapter-json.md).


> Json library is not explicitly required, you need to make sure it is available in your composer.json file.
Expand Down Expand Up @@ -66,7 +67,6 @@ data_frame()

It is also possible to export the rows using the [json lines](https://jsonlines.org/) format


```php
<?php

Expand All @@ -84,3 +84,67 @@ data_frame()
->write(to_json_lines(\sys_get_temp_dir() . '/file.jsonl'))
->run();
```

## JSON Schema conversion

The adapter can convert [JSON Schema](https://json-schema.org) documents into Flow schemas and back.

```php
<?php

use function Flow\ETL\Adapter\JSON\{from_json, schema_from_json_schema, schema_to_json_schema};
use function Flow\ETL\DSL\data_frame;
use function Flow\Filesystem\DSL\path;

$schema = schema_from_json_schema(path(__DIR__ . '/schemas/person.json'));

data_frame()
->read(from_json(__DIR__ . '/people.json', schema: $schema))
->collect()
->run();

$jsonSchema = schema_to_json_schema($schema); // draft 2020-12 document as array
```

`schema_from_json_schema()` accepts a decoded document (array), a raw JSON document (string) or a `Path` to a schema
file.

### Type mapping

| JSON Schema | Flow |
|------------------------------------------------------------------------------|-------------------------------------------------------------------|
| `string` (+ `format: date` / `date-time` / `time` / `uuid` / `html` / `xml`) | `string` / `date` / `datetime` / `time` / `uuid` / `html` / `xml` |
| `integer` / `number` / `boolean` / `null` | `integer` / `float` / `boolean` / null column |
| `array` + `items` | `list<items>` |
| `object` + `properties` | `structure` (`required` controls optional members) |
| `object` + `additionalProperties: <schema>` | `map<string, value>` |
| `type: ["object", "array"]` | `json` |
| empty schema `{}` / boolean schema `true` | `json` (accept anything, marked in metadata) |
| `enum` / `const` | scalar type derived from the values, values preserved in metadata |
| `type: [X, "null"]`, `anyOf`/`oneOf` with a null member | nullable X |
| heterogeneous `type` arrays, `anyOf`, `oneOf` | union type |
| `allOf` | deep merged object schema |

Annotations (`description`, `title`, `default`, `examples`, constraints like `pattern` or `minimum`) of top level
properties are preserved in definition metadata under `json_schema.*` keys and re-emitted by `schema_to_json_schema()`.

### References

`$ref` is resolved automatically:

- internal pointers (`#/$defs/...`, legacy `#/definitions/...`),
- relative file paths (resolved against the schema file location, requires loading the schema from a `Path`),
- remote `http(s)://` documents - requires passing a PSR-18 client and a PSR-17 request factory to
`schema_from_json_schema()`.

### Limitations

- Recursive schemas cannot be represented by a Flow schema, circular references throw `CircularReferenceException`.
- `$anchor`, `not`, `if`/`then`/`else`, `contains`, `patternProperties`, `unevaluatedProperties`, `unevaluatedItems`,
`dependentSchemas`, `dependentRequired`, `$dynamicRef`, `$dynamicAnchor` and `propertyNames` are not supported and
throw `UnsupportedKeywordException`.
- References are inlined during conversion, `schema_to_json_schema()` does not reconstruct `$ref` structures.
- Annotations of nested (non top level) properties are not preserved.
- Accept-anything properties are emitted as the boolean schema `true` (an empty PHP array would serialize to a JSON
list); the boolean schema `false` (reject everything) cannot be represented and throws.
- An empty Flow schema cannot be converted to JSON Schema and throws.
5 changes: 4 additions & 1 deletion src/adapter/etl-adapter-json/composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@
"php": "~8.3.0 || ~8.4.0 || ~8.5.0",
"ext-json": "*",
"flow-php/etl": "self.version",
"halaxa/json-machine": "^1.1"
"flow-php/filesystem": "self.version",
"halaxa/json-machine": "^1.1",
"psr/http-client": "^1.0",
"psr/http-factory": "^1.0"
},
"config": {
"optimize-autoloader": true,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
<?php

declare(strict_types=1);

namespace Flow\ETL\Adapter\JSON\JsonSchema\Exception;

use Flow\ETL\Exception\RuntimeException;

use function implode;
use function sprintf;

final class CircularReferenceException extends RuntimeException
{
/**
* @param array<string> $chain
*/
public function __construct(string $ref, array $chain)
{
parent::__construct(sprintf(
'Circular reference detected: "%s", resolution chain: %s',
$ref,
implode(' -> ', $chain),
));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<?php

declare(strict_types=1);

namespace Flow\ETL\Adapter\JSON\JsonSchema\Exception;

use Flow\ETL\Exception\RuntimeException;

use function sprintf;

final class UnresolvableReferenceException extends RuntimeException
{
public function __construct(string $ref, string $reason)
{
parent::__construct(sprintf('Cannot resolve reference "%s": %s', $ref, $reason));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<?php

declare(strict_types=1);

namespace Flow\ETL\Adapter\JSON\JsonSchema\Exception;

use Flow\ETL\Exception\RuntimeException;

use function sprintf;

final class UnsupportedKeywordException extends RuntimeException
{
public function __construct(string $keyword, string $path)
{
parent::__construct(sprintf(
'JSON Schema keyword "%s" at path "%s" is not supported by the Flow schema converter',
$keyword,
$path,
));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
<?php

declare(strict_types=1);

namespace Flow\ETL\Adapter\JSON\JsonSchema;

use function strlen;
use function substr;

enum JsonSchemaMetadata: string
{
case ANY = 'json_schema.any';
case DEFAULT = 'json_schema.default';
case DESCRIPTION = 'json_schema.description';
case ENUM = 'json_schema.enum';
case EXAMPLES = 'json_schema.examples';
case EXCLUSIVE_MAXIMUM = 'json_schema.exclusiveMaximum';
case EXCLUSIVE_MINIMUM = 'json_schema.exclusiveMinimum';
case FORMAT = 'json_schema.format';
case MAXIMUM = 'json_schema.maximum';
case MAX_ITEMS = 'json_schema.maxItems';
case MAX_LENGTH = 'json_schema.maxLength';
case MINIMUM = 'json_schema.minimum';
case MIN_ITEMS = 'json_schema.minItems';
case MIN_LENGTH = 'json_schema.minLength';
case PATTERN = 'json_schema.pattern';
case PREFIX_ITEMS = 'json_schema.prefixItems';
case TITLE = 'json_schema.title';

public function keyword(): string
{
return substr($this->value, strlen('json_schema.'));
}
}
Loading
Loading