# Cleaning up resources Commerce Layer lets you remove resources and their associated relationships in batches. To do that, you need to create a new [cleanup](https://app.gitbook.com/s/RWJeylueWkzLadK710XZ/cleanups) resource and specify the `resource_type` you want to remove. Optionally, you can also apply some [filters](#filtering-the-data-to-be-removed) to narrow the affected data. The process is **asynchronous** and you can poll the `status` attribute to check the clean-up progress. Once the operation has been completed you can check the number of removed items by inspecting the `processed_count` attribute. In case of [validation errors](#handling-cleanup-errors), the `errors_count` and `errors_log` attributes are populated. {% hint style="danger" %} Since it is fired in the background as soon as the cleanup resource is created, **the removal process** i**s not reversible**: all of the resources matching the filters criteria and their associated relationships will be removed in bulk. We suggest that you create a backup of the resources you're going to clean up [exporting](https://docs.commercelayer.io/core/exporting-resources) them first (making sure to use the same [filters](https://docs.commercelayer.io/core/exporting-resources#filtering-exported-data) and [include](https://docs.commercelayer.io/core/exporting-resources#including-associations) all of the affected relationships). {% endhint %} ### Cleanup limits #### Maximum cleanup size There is no limit on the total number of resources you can clean up, but the single batches are subject to some soft limits: the `records_count` must be a maximum of **10000** records, otherwise the cleanup will be rejected at the time of creation. #### Maximum error percentage During the clean-up process, the errors count is also monitored: cleanups whose `errors_count` attribute value overflows the maximum percentage of **10%** of the total `records_count` will be interrupted. #### Concurrent cleanups The maximum number of concurrent cleanups (i.e. cleanups whose status is `pending` or `in_progess`) allowed per organization is **10**. {% hint style="info" %} If you absolutely need to clean up any of the [supported resources](#supported-resources) in one go, overriding the maximum cleanup size and concurrent cleanup API limits above, you can leverage the power of our [CLI cleanups plugin](https://github.com/commercelayer/commercelayer-cli-plugin-cleanups/) ([see example](#cleaning-up-more-than-10k-skus-using-the-cli)). {% endhint %} ### Supported resources At moment, you can clean up the following resources and associated relationships (more to come): * **Bundles** * **Gift cards** * **Prices** > price tiers * **Promotions** > promotion rules * **SKU lists** > SKU list items, bundles * **SKU options** > line item options * **SKUs** > in-stock subscriptions, prices, SKU list items, stock items, tax categories * **Stock items** Please find some examples of how to clean up them [here below](#examples). {% hint style="info" %} Please note that associated relationships will be **automatically removed** along with the parent resource. The clean-up process works in cascade (e.g. removing a promotion will also remove its associated promotion rules and all the related coupons). {% endhint %} ### Constraints In order to preserve data integrity the cleanups are subject to some relationship constraints. If such constraints are violated an error is added to the `errors_log` and the `errors_count` is incremented (resulting in an interruption if the [10% threshold](#maximum-error-percentage) is exceeded): * **SKU lists** — must not be linked to any promotion or promotion rules (i.e. any associated promotion must be removed first) * **SKUs** — must not be linked to any SKU list item within a bundle (i.e. any associated bundle must be removed first) ### Filtering the data to be removed When cleaning up resources, you can fine-tune the data to be removed by applying some filters (both to the resources and their relationships) using the `filters` attribute: ``` ... "filters": { "{{predicate}}": {{value}}, ... } ``` To compose the filter predicate you just need to follow the [same syntax](https://docs.commercelayer.io/core/filtering-data) you use when filtering a collection of resources — `{{attributes}}_{{matcher}}`. You must specify filtering rules as a valid JSON object. List values for the `*_in` matcher need to be expressed as arrays. {% hint style="warning" %} When specifying `filters` as a JSON attribute, the syntax of the `*_jcont` predicate changes. You must specify the value of the predicate as a plain string: `{ "metadata_jcont": \"{"key": "value"}\" }` {% endhint %} {% content-ref url="filtering-data" %} [filtering-data](https://docs.commercelayer.io/core/filtering-data) {% endcontent-ref %} ### Examples #### Cleaning up a batch of SKUs {% tabs %} {% tab title="Request" %} The following request creates a cleanup to remove the SKUs matching the specified code, along with their relationships: ```shell curl -g -X POST \ 'https://yourdomain.commercelayer.io/api/cleanups' \ -H 'Accept: application/vnd.api+json' \ -H 'Authorization: Bearer your-access-token' \ -H 'Content-Type: application/vnd.api+json' \ -d '{ "data": { "type": "cleanups", "attributes": { "resource_type": "skus", "filters": { "code_start": "SHIRT" } } } }' ``` {% endtab %} {% tab title="Response" %} On success, the API responds with a `201 Created` status code, returning the created cleanup object: ```json { "data": { "id": "WmjloIJzAS", "type": "cleanups", "links": {...}, "attributes": { "resource_type": "skus", "status": "pending", "started_at": null, "completed_at": null, "interrupted_at": null, "filters": {"code_start": "SHIRT"}, "records_count": 100, "errors_count": 0, "processed_count": 0, "errors_log": {...}, "created_at": "2018-01-01T12:00:00.000Z", "updated_at": "2018-01-01T12:00:00.000Z", "reference": null, "reference_origin": null, "metadata": {} }, "relationships": { "events": { ... } }, "meta": {...} } } ``` {% endtab %} {% endtabs %} #### Cleaning up a batch of promotions {% tabs %} {% tab title="Request" %} The following request creates a cleanup to remove all the promotions that expired after a specified date, along with their relationships: ```shell curl -g -X POST \ 'https://yourdomain.commercelayer.io/api/cleanups' \ -H 'Accept: application/vnd.api+json' \ -H 'Authorization: Bearer your-access-token' \ -H 'Content-Type: application/vnd.api+json' \ -d '{ "data": { "type": "cleanups", "attributes": { "resource_type": "promotions", "filters": { "expired_at_gteq": "2018-01-01T12:00:00.000Z" } } } }' ``` {% endtab %} {% tab title="Response" %} On success, the API responds with a `201 Created` status code, returning the created import object: ```json { "data": { "id": "HGfdRsErUk", "type": "cleanups", "links": {...}, "attributes": { "resource_type": "promotions", "status": "pending", "started_at": null, "completed_at": null, "interrupted_at": null, "filters": {"expired_at_gteq": "2018-01-01T12:00:00.000Z"}, "records_count": 55, "errors_count": 0, "processed_count": 0, "errors_log": {...}, "created_at": "2018-01-01T12:00:00.000Z", "updated_at": "2018-01-01T12:00:00.000Z", "reference": null, "reference_origin": null, "metadata": {} }, "relationships": { "events": { ... } }, "meta": {...} } } ``` {% endtab %} {% endtabs %} #### Cleaning up more than 10K SKUs using the CLI {% tabs %} {% tab title="CLI command" %} The following command cleans up all the SKUs whose code contains a specific subset of characters: ``` commercelayer cleanups:create -t skus -w code_cont=FLASHSALE ``` {% endtab %} {% tab title="Success message" %} On success, the CLI prompts this message on your terminal, separating the single cleanups and showing some basic information about the completed processes: 
|     ID     |     Cleanup progress      |   %  |    Status   | TBP↓  | Prc.  | Err.  |
---------------------------------------------------------------------------------------
| DRlYbHDopv | █████████████████████████ | 100% | completed   |     0 | 10000 |     0 | 
| MReZgHXLpw | █████████████████████████ | 100% | completed   |     0 |  5000 |     0 | 

Cleanup of 15000 skus completed.
{% endtab %} {% endtabs %} ### Checking the cleanup status You can inspect the status of a specific cleanup by fetching it by ID and looking at the `status` attribute. {% tabs %} {% tab title="Request" %} The following request fetches a single cleanup, the one identified by the ID "WmjloIJzAS": ```shell curl -g -X GET \ 'https://yourdomain.commercelayer.io/api/cleanups/WmjloIJzAS' \ -H 'Content-Type: application/vnd.api+json' \ -H 'Authorization: Bearer your-access-token' ``` {% endtab %} {% tab title="Response" %} On success, the API responds with a `200 OK` status code, returning the single cleanup object: ```json { "data": { "id": "WmjloIJzAS", "type": "cleanups", "links": {...}, "attributes": { "resource_type": "skus", "status": "completed", "started_at": "2018-01-01T12:00:00.000Z", "completed_at": "2018-01-01T12:01:00.000Z", "interrupted_at": null, "filters": {}, "records_count": 100, "errors_count": 1, "processed_count": 99, "errors_log": {...}, "created_at": "2018-01-01T12:00:00.000Z", "updated_at": "2018-01-01T12:00:00.000Z", "reference": null, "reference_origin": null, "metadata": {} }, "relationships": { "events": { ... } }, "meta": {...} } } ``` {% endtab %} {% endtabs %} {% hint style="info" %} When you create a cleanup, it tries immediately to start the removal process, entering the `in_progress` status. In case the async queue is saturated, the cleanup remains `pending` until it gets a chance to be processed. {% endhint %} {% hint style="warning" %} If a cleanup gets stuck in the `in_progress` status for any reason, you can mark it as `interrupted` by patching it with the `_interrupt` trigger attribute set to `true`. {% endhint %} ### Webhooks for cleanups You can also leverage Commerce Layer real-time webhooks mechanism, listen to `cleanups.create`, `cleanups.start`, `cleanups.complete`, `cleanups.interrupt`, or `cleanups.destroy` and react properly. {% content-ref url="real-time-webhooks" %} [real-time-webhooks](https://docs.commercelayer.io/core/real-time-webhooks) {% endcontent-ref %} ### Handling cleanup errors When a cleanup is completed or interrupted, [constraints](#constraints) errors are reported through the `errors_count` and `errors_log` attributes: ```json ... "errors_count": 5, "processed_count": 95, "errors_log": { "HjKhdjGfDD": "Cannot delete record because of dependent bundles", ... }, ... ```