<jupyter_start><jupyter_text>File datasetsIf there's a particular data type you are working with, it's usually better to use the type-specific dataset object. However, there are still a few core operations that can be performed on generic files.  Initializing datasetsLet's start by creating a simple dataset of text files.<jupyter_code>import os
outdir = "toy_files"
os.mkdir(outdir)

file1 = "file1_contents"
file2 = "file2_contents"
file2_duplicate = "file2_contents"
file3 = "file3_contents"
with open(outdir + "/1.file", "w") as fd:
    fd.write(file1)
with open(outdir + "/2.file", "w") as fd:
    fd.write(file2)
with open(outdir + "/2_dup.file", "w") as fd:
    fd.write(file2_duplicate)
with open(outdir + "/3.file", "w") as fd:
    fd.write(file3)
from kaishi.core.dataset import FileDataset
fd = FileDataset(outdir)
fd.file_report()<jupyter_output>Current file list:
+-------+------------+------------------------+--------+
| Index | File Name  | Children               | Labels |
+-------+------------+------------------------+--------+
|   0   |   3.file   |     {'duplicates': []} |   []   |
|   1   |   2.file   |     {'duplicates': []} |   []   |
|   2   | 2_dup.file |     {'duplicates': []} |   []   |
|   3   |   1.file   |     {'duplicates': []} |   []   |
+-------+------------+------------------------+--------+
Filtered files:
+-----------+---------------+
| File Name | Filter Reason |
+-----------+---------------+
+-----------+---------------+<jupyter_text>File procesing pipelinesThere are fewer components available for files compared to other types, as the other types inherit from the FileGroup class. However, there are still plenty of options available to perform some common operations.<jupyter_code>fd.get_pipeline_options()
fd.configure_pipeline(["FilterDuplicateFiles", "FilterSubsample"])
print(fd.pipeline)
fd.pipeline.components[1].configure(N=2, seed=42)
print(fd.pipeline)
fd.run_pipeline()
fd.file_report()<jupyter_output>Current file list:
+-------+-----------+----------------------------------+--------+
| Index | File Name | Children                         | Labels |
+-------+-----------+----------------------------------+--------+
|   0   |   3.file  |     {'duplicates': []}           |   []   |
|   1   |   2.file  |     {'duplicates': [2_dup.file]} |   []   |
+-------+-----------+----------------------------------+--------+
Filtered files:
+------------+---------------+
| File Name  | Filter Reason |
+------------+---------------+
| 2_dup.file |   duplicates  |
|   1.file   |   subsample   |
+------------+---------------+
StarcoderdataJupyterStructuredCleanDedup