Purge policies
The Purge Policies module allows you to create instance/company purge policies to archive or purge ARCHIVE data or USERDATA automatically given a certain condition.
The Purge Policies module includes the following sections:
- Instance Purge Settings: Allows you to add ARCHIVE or USERDATA purge policies for the instance level. You can set up a policy action to purge or archive data and limit the purge by the number of periods, by the project's data size, or by the free-disk-space percentage.
- Companies Purge Settings: Allows you to add ARCHIVE or USERDATA purge policies for a company. You can set up a policy action to purge or archive data and limit the purge by the number of periods, by the project's data size, or by the free-disk-space percentage.
You can also filter the instance/company purge policies list from this sub-module.
Archiving needs additional disk space
When data is archived, the data is extracted from a Clickhouse Data Server and stored as a .csv.gz file. Each .csv.gz file is up to ten times the size of the data size displayed in Litmus Edge Manager (The data size that is shown in the Stats By Periods section of Statistics is a compressed size).
Before you archive, ensure the Litmus Edge Manager has enough disk space to store all the .csv.gz file data that it will archive. Insufficient disk space will prevent the data from being archived. After archiving, review the list of archive files to ensure the archive operation was successful.
Access purge policies
To access the purge policies:
- Log in to the Litmus Edge Manager (LEM) Admin Console at the following URL: https://[LEM IP address]:8446.
- From the Navigation panel, select Data Lifecycle. Data Lifecycle's Statistics pane by default appears.
- From Data Lifecycle's navigation sub-panel, select Purge Policies. Data Lifecycle's Purge Policies pane appears.
Refer to the following actions you can take on the Litmus Edge Manager Admin Console Data Lifecycle's Purge Policies pane.
Action | Details |
|---|---|
[1] Show filter / Hide filter | Search by Data Type and Policy Action |
[2] Create Instance Purge Policy | Option availability depends on previous selections. See the Create a Policy section below. |
[3] Action menu |
|
[4] Show filter / Hide filter |
|
[5] Create Company Purge Policy | Option availability depends on previous selections. |
[6] Action menu |
|
Period explained
Litmus Edge Manager keeps track of disk usage in partitions of months. The tracked disk usage is only for Clickhouse data (hence all archive/purge operations are applicable for only Clickhouse data). The disk usage amount is calculated either from the instance or each individual project. Each month of disk usage is defined as a period.
For example, your Litmus Edge Manager has five projects. To calculate disk usage, each project will be categorized as one period for the same month. So Litmus Edge Manager will generate five periods for the same month, each period including one project.
As an additional example, three months have passed. For the five projects above, each of them experienced some disk usage in those three months. So you will have 15 additional periods (five projects of disk usage * 3 months).
Periods are also categorized as previous or current. If the current system date is March 2023, then all disk usage by projects during March 2023 are current periods. All periods from February 2023 and before are previous periods.
Create policy
A policy is only applicable for a previous period. See Statistics.
Note: When policy types are considered, disk usage for a current period of the Litmus Edge Manager will be ignored.
- Select Company: Select a company to apply to policy towards. All of the company's projects and the projects' respective periods will be included when creating a policy. This option is only available for Companies Purge Settings.
- Select Data Object:
- Archive: Apply the policy toward an existing archive in Management.
- Userdata: Apply the policy towards existing data.
Note:
- For Instance Purge Settings, you may only create one policy per Data Object.
- For Companies Purge Settings, you may only create one policy per Company-Data Object pairing.
- Policy Action:
- Archive: Archive the data if a previous period fails to stay within a policy type limit. You cannot archive an Archive Data object. When this policy is selected, archiving a period will also automatically purge the data after successful archiving.
- Purge: Delete the data if a previous period fails to stay within a policy type limit.
- Policy Type (See the More on Policy Types section below)
- Limit by number of periods: You cannot select this for an Archive Data data object.
- Limit by project's data size
- Free disk space limit, %
- Policy Limits Value
- Limit by number of periods: At least 1 or more.
- Limit by project's data size: : At least 1 or more.
- Free disk space limit, %: As of Litmus Edge Manager 2.10.1, enter a percentage value between 0-50%. Decimal values are allowed.
More on policy types
Refer to the Period Explained section above to learn more about the definition of period in Litmus Edge Manager's Data Lifecycle feature.
When a policy is selected for instance purge settings, it will check all existing project's previous periods.
When a policy is selected for company purge settings, it will check the selected company's existing project's previous periods.
Limit by number of periods
Data will be archived/purged based on the number of months you enter.
Example 1
You enter 1 and the current date is January 1, 2023.
- Litmus Edge Manager checks the age (in months) of every single previous period. So December 2022 would be 1, November 2022 would be 2, and so on.
- It compares the period's age against the age you entered earlier (1). If the period's age is equal to or greater than the age you entered earlier (1), then archive/purge.
- All data within each previous individual month will be archived/purged starting backwards from December 2022.
- When the current date advances to February 1, 2023, the data from January 2023 will be archived/purged.
Example 2
You enter 3 and the current date is January 1, 2023.
- Litmus Edge Manager checks the age (in months) of every single previous period. So December 2022 would be 1, November 2022 would be 2, and so on.
- It compares the period's age against the age you entered earlier (3). If the period's age is equal to or greater than the age you entered earlier (3), then archive/purge.
- All data within each previous individual month will be archived/purged starting backwards from October 2022.
- When the current date advances to February 2023, the data from November 2022 will be archived/purged. This means the three-month duration of December 2022-February 2023 is stored in Litmus Edge Manager.
Limit by project's data size
Data will be archived/purged for a previous period if the data size is equal to or greater than the data size you enter. The oldest data is archived/purged first.
Example 1
You enter 1GB and the current date is January 1, 2023.
- Litmus Edge Manager checks the data size of every single previous period.
- If the single previous period's data size is 1GB or more, it will be archived/purged.
- When the current date advances to February 1, 2023, Litmus Edge Manager checks for all the data size for the January 2023 period. All January 2023 periods with a data size of 1GB or more are archived/purged.
Free disk space limit
Data will be archived/purged once as long as the free disk-space percentage is below the specified threshold. The oldest data is archived/purged first.
Example 1
You enter 1% and the total disk space size is 100GB.
- Once Litmus Edge Manager's free disk space is less than 1GB, it will archive/purge all previous periods.
- If Litmus Edge Manager's free disk space is 1GB or more, do nothing.
- For purging, all the purge operations of each period will happen concurrently. Therefore, if a purge increases the free disk space percentage above the 1GB threshold, it will not stop the purge for the remaining periods.
Example 2
You enter 50% and the total disk space size is 100GB.
- Once Litmus Edge Manager's free disk space is less than 50GB, it will archive/purge all previous periods.
- If Litmus Edge Manager's free disk space is 50GB or more, do nothing.
- For purging, all the purge operations of each period will happen concurrently. Therefore, if a purge increases the free disk space percentage above the 50GB threshold, it will not stop the purge for the remaining periods.