Load: Fix active load reliability issues#18224
Conversation
| }); | ||
| } | ||
| } catch (final IOException | RuntimeException e) { | ||
| org.apache.iotdb.commons.utils.FileUtils.deleteFileOrDirectory(transferDir, true); |
There was a problem hiding this comment.
Updated the failure path to retry deletion of an existing partial transfer directory, so transient deletion failures are less likely to leave an incomplete handoff behind. Thanks for pointing this out.
| private static boolean isTsFile(final File file) { | ||
| return file.getAbsolutePath().equals(getTsFilePath(file.getAbsolutePath())); | ||
| } |
There was a problem hiding this comment.
Changed this to file.getName().endsWith(TsFileConstant.TSFILE_SUFFIX). This is clearer and restricts the terminal publish ordering to actual TsFiles.
| final List<File> sourceFiles = createTsFileAndCompanions(); | ||
| for (final File sourceFile : sourceFiles) { | ||
| Files.write( | ||
| new File(targetDir, sourceFile.getName()).toPath(), | ||
| "existing".getBytes(StandardCharsets.UTF_8)); | ||
| } | ||
|
|
||
| LoadUtil.transferFilesToActiveDir(targetDir, sourceFiles, true); | ||
|
|
||
| final File[] transferDirs = targetDir.listFiles(File::isDirectory); | ||
| Assert.assertNotNull(transferDirs); | ||
| Assert.assertEquals(1, transferDirs.length); | ||
| for (final File sourceFile : sourceFiles) { | ||
| Assert.assertFalse(sourceFile.exists()); | ||
| final File transferredFile = new File(transferDirs[0], sourceFile.getName()); | ||
| Assert.assertTrue(transferredFile.exists()); | ||
| Assert.assertArrayEquals( | ||
| sourceFile.getName().getBytes(StandardCharsets.UTF_8), | ||
| Files.readAllBytes(transferredFile.toPath())); | ||
| Assert.assertArrayEquals( | ||
| "existing".getBytes(StandardCharsets.UTF_8), | ||
| Files.readAllBytes(new File(targetDir, sourceFile.getName()).toPath())); | ||
| } |
There was a problem hiding this comment.
What is the point of the "existing" file?
There was a problem hiding this comment.
Reworked the test instead of keeping the synthetic existing files. It now performs two handoffs of same-named groups with different contents, then verifies that they are placed in two isolated transfer directories and that each directory contains companions from the same handoff.
2 participants
Description
Active load handoff
Scanner and loader lifecycle
Load resource cleanup
Test coverage
This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR