Cleaning up resources

How to bulk remove resources and their relationships

Commerce Layer lets you remove resources and their associated relationships in batches. To do that, you need to create a new cleanup resource and specify the resource_type you want to remove. Optionally, you can also apply some filters to narrow the affected data.

The process is asynchronous and you can poll the status attribute to check the clean-up progress. Once the operation has been completed you can check the number of removed items by inspecting the processed_count attribute. In case of validation errors, the errors_count and errors_log attributes are populated.

Since it is fired in the background as soon as the cleanup resource is created, the removal process is not reversible: all of the resources matching the filters criteria and their associated relationships will be removed in bulk. We suggest that you create a backup of the resources you're going to clean up exporting them first (making sure to use the same filters and include all of the affected relationships).

Cleanup limits

Maximum cleanup size

There is no limit on the total number of resources you can clean up, but the single batches are subject to some soft limits: the records_count must be a maximum of 10000 records, otherwise the cleanup will be rejected at the time of creation.

Maximum error percentage

During the clean-up process, the errors count is also monitored: cleanups whose errors_count attribute value overflows the maximum percentage of 10% of the total records_count will be interrupted.

Concurrent cleanups

The maximum number of concurrent cleanups (i.e. cleanups whose status is pending or in_progess) allowed per organization is 10.

If you absolutely need to clean up any of the supported resources in one go, overriding the maximum cleanup size and concurrent cleanup API limits above, you can leverage the power of our CLI cleanups plugin (see example).

Supported resources

At moment, you can clean up the following resources and associated relationships (more to come):

• Bundles

• Gift cards

• Prices > price tiers

• Promotions > promotion rules

• SKU lists > SKU list items, bundles

• SKU options > line item options

• SKUs > in-stock subscriptions, prices, SKU list items, stock items, tax categories

• Stock items

Please find some examples of how to clean up them here below.

Please note that associated relationships will be automatically removed along with the parent resource. The clean-up process works in cascade (e.g. removing a promotion will also remove its associated promotion rules and all the related coupons).

Constraints

In order to preserve data integrity the cleanups are subject to some relationship constraints. If such constraints are violated an error is added to the errors_log and the errors_count is incremented (resulting in an interruption if the 10% threshold is exceeded):

• SKU lists — must not be linked to any promotion or promotion rules (i.e. any associated promotion must be removed first)

• SKUs — must not be linked to any SKU list item within a bundle (i.e. any associated bundle must be removed first)

Filtering the data to be removed

When cleaning up resources, you can fine-tune the data to be removed by applying some filters (both to the resources and their relationships) using the filters attribute:

...
  "filters": {
    "{{predicate}}": {{value}},
    ...
  }

To compose the filter predicate you just need to follow the same syntax you use when filtering a collection of resources — {{attributes}}_{{matcher}}. You must specify filtering rules as a valid JSON object. List values for the *_in matcher need to be expressed as arrays.

pageFiltering data

Examples

Cleaning up a batch of SKUs

Request

The following request creates a cleanup to remove the SKUs matching the specified code, along with their relationships:

curl -g -X POST \
  'https://yourdomain.commercelayer.io/api/cleanups' \
  -H 'Accept: application/vnd.api+json' \
  -H 'Authorization: Bearer your-access-token' \
  -H 'Content-Type: application/vnd.api+json' \
  -d '{
	"data": {
	  "type": "cleanups",
	  "attributes": {
	    "resource_type": "skus",
            "filters": {
	      "code_start": "SHIRT"
	    }
	  }
	}
      }'

Response

On success, the API responds with a 201 Created status code, returning the created cleanup object:

{
  "data": {
    "id": "WmjloIJzAS",
    "type": "cleanups",
    "links": {...},
    "attributes": {
      "resource_type": "skus",
      "status": "pending",
      "started_at": null,
      "completed_at": null,
      "interrupted_at": null,
      "filters": {"code_start": "SHIRT"},
      "records_count": 100,
      "errors_count": 0,
      "processed_count": 0,
      "errors_log": {...},
      "created_at": "2018-01-01T12:00:00.000Z",
      "updated_at": "2018-01-01T12:00:00.000Z",
      "reference": null,
      "reference_origin": null,
      "metadata": {}
    },
    "relationships": {
      "events": { ... }
    },
    "meta": {...}
  }
}

Cleaning up a batch of promotions

Request

The following request creates a cleanup to remove all the promotions that expired after a specified date, along with their relationships:

curl -g -X POST \
  'https://yourdomain.commercelayer.io/api/cleanups' \
  -H 'Accept: application/vnd.api+json' \
  -H 'Authorization: Bearer your-access-token' \
  -H 'Content-Type: application/vnd.api+json' \
  -d '{
	"data": {
	  "type": "cleanups",
	  "attributes": {
	    "resource_type": "promotions",
	    "filters": {
	      "expired_at_gteq": "2018-01-01T12:00:00.000Z"
	    }
	  }
	}
      }'

Response

On success, the API responds with a 201 Created status code, returning the created import object:

{
  "data": {
    "id": "HGfdRsErUk",
    "type": "cleanups",
    "links": {...},
    "attributes": {
      "resource_type": "promotions",
      "status": "pending",
      "started_at": null,
      "completed_at": null,
      "interrupted_at": null,
      "filters": {"expired_at_gteq": "2018-01-01T12:00:00.000Z"},
      "records_count": 55,
      "errors_count": 0,
      "processed_count": 0,
      "errors_log": {...},
      "created_at": "2018-01-01T12:00:00.000Z",
      "updated_at": "2018-01-01T12:00:00.000Z",
      "reference": null,
      "reference_origin": null,
      "metadata": {}
    },
    "relationships": {
      "events": { ... }
    },
    "meta": {...}
  }
}

Cleaning up more than 10K SKUs using the CLI

CLI command

The following command cleans up all the SKUs whose code contains a specific subset of characters:

commercelayer cleanups:create -t skus -w code_cont=FLASHSALE

Success message

On success, the CLI prompts this message on your terminal, separating the single cleanups and showing some basic information about the completed processes:

|     ID     |     Cleanup progress      |   %  |    Status   | TBP↓  | Prc.  | Err.  |
---------------------------------------------------------------------------------------
| DRlYbHDopv | █████████████████████████ | 100% | completed   |     0 | 10000 |     0 | 
| MReZgHXLpw | █████████████████████████ | 100% | completed   |     0 |  5000 |     0 | 

Cleanup of 15000 skus completed.

Checking the cleanup status

You can inspect the status of a specific cleanup by fetching it by ID and looking at the status attribute.

When you create a cleanup, it tries immediately to start the removal process, entering the in_progress status. In case the async queue is saturated, the cleanup remains pending until it gets a chance to be processed.

Request

The following request fetches a single cleanup, the one identified by the ID "WmjloIJzAS":

curl -g -X GET \ 
  'https://yourdomain.commercelayer.io/api/cleanups/WmjloIJzAS' \
  -H 'Content-Type: application/vnd.api+json' \
  -H 'Authorization: Bearer your-access-token'

Response

On success, the API responds with a 200 OK status code, returning the single cleanup object:

{
  "data": {
    "id": "WmjloIJzAS",
    "type": "cleanups",
    "links": {...},
    "attributes": {
      "resource_type": "skus",
      "status": "completed",
      "started_at": "2018-01-01T12:00:00.000Z",
      "completed_at": "2018-01-01T12:01:00.000Z",
      "interrupted_at": null,
      "filters": {},
      "records_count": 100,
      "errors_count": 1,
      "processed_count": 99,
      "errors_log": {...},
      "created_at": "2018-01-01T12:00:00.000Z",
      "updated_at": "2018-01-01T12:00:00.000Z",
      "reference": null,
      "reference_origin": null,
      "metadata": {}
    },
    "relationships": {
      "events": { ... }
    },
    "meta": {...}
  }
}

You can also leverage Commerce Layer real-time webhooks mechanism, listen to cleanups.create, cleanups.start, cleanups.complete, cleanups.interrupt, or cleanups.destroy and react properly.

pageReal-time webhooks

Handling cleanup errors

When a cleanup is completed or interrupted, constraints errors are reported through the errors_count and errors_log attributes:

...

"errors_count": 5,
"processed_count": 95,
"errors_log": {
  "HjKhdjGfDD": "Cannot delete record because of dependent bundles",
  ...
},

...