Raincloud/ProjectStructure_HOME
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

HOME config

Browse Source
This commit is contained in:
Raincloud
2025-11-11 17:57:28 -07:00
parent 6d4cdb0549
commit e1c1aacd0d
2 changed files with 330 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

329
.specstory/history/2025-11-11_01-50Z-update-configloader-to-overwrite-gitattributes-and-gitignore.md
View File

@@ -4560,6 +4560,300 @@ Lines added: 1, lines removed: 1
---
<tool-use data-tool-type="generic" data-tool-name="read_lints">
<details>
<summary>Tool use: **read_lints** • Read lints for 1 file</summary>
Lint paths:
- `zip_sequences.py`
Lint results
Lint errors for `zip_sequences.py`
| Message | Range | Severity |
|-------|-------|----------|
| Variable "source" is not accessed | 73:9-73:15 | DIAGNOSTIC_SEVERITY_HINT |
| "seq_dir" is not accessed | 186:26-186:33 | DIAGNOSTIC_SEVERITY_HINT |
| Variable "zip_path" is not accessed | 326:18-326:26 | DIAGNOSTIC_SEVERITY_HINT |
| Variable "state_path" is not accessed | 326:28-326:38 | DIAGNOSTIC_SEVERITY_HINT |
</details>
</tool-use>
---
---
Updated the code to use `0` for auto-calculate instead of `null`:
1. **DEFAULT_CONFIG**: Changed from `None` to `0`
2. **Logic**: Treats `0` as `None` (auto-calculate), so `0` triggers RAM-based calculation
3. **config.json**: Set to `0`
Now:
- `"Max7zInst": 0` = auto-calculate based on RAM
- `"Max7zInst": 1` = limit to 1 instance
- `"Max7zInst": 2` = limit to 2 instances
- etc.
This makes the config clearer: `0` means automatic, and any positive number sets a hard limit.
---
_**User (2025-11-11 01:50Z)**_
I think we're just dealing with massive sequences. This is only 1 instance, so even if I had 2x the ram, I would only be able to afford 2!
---
_**Agent (model default, mode Agent)**_
---
Large sequences are using far more RAM than estimated. Making the dictionary size configurable and reducing the default to lower per-instance usage:
---
<tool-use data-tool-type="read" data-tool-name="read_file">
<details><summary>Tool use: **read_file** • Read file: zip_sequences.py</summary>
</details>
</tool-use>
---
<tool-use data-tool-type="read" data-tool-name="read_file">
<details><summary>Tool use: **read_file** • Read file: zip_sequences.py</summary>
</details>
</tool-use>
---
<tool-use data-tool-type="read" data-tool-name="read_file">
<details><summary>Tool use: **read_file** • Read file: config.json</summary>
</details>
</tool-use>
---
---
_**User (2025-11-11 01:50Z)**_
Nope, we're keeping the dictsize as large as possible. I just need to keep in mind my limitations on a per-project basis.
---
_**Agent (model default, mode Agent)**_
---
The current setup supports this. Keep the dictionary size at 1GB for best compression, and set `Max7zInst` per project in `.config/config.json` to limit concurrent instances based on your RAM.
For example:
- `"Max7zInst": 1` - Only 1 concurrent 7z instance (safest for large sequences)
- `"Max7zInst": 2` - 2 concurrent instances (if you have more RAM)
- `"Max7zInst": 0` - Auto-calculate (may still hit limits with very large sequences)
The dictionary size stays at 1GB for maximum compression, and you control concurrency per project. This lets you tune each project based on sequence sizes and available RAM.
---
_**User (2025-11-11 06:13Z)**_
this is on another system. ignore the 7z PATH error. Are there any other relevant errors?
& "R:\Creative\artsy\maya\2025-02 BondingArtifact\Renders\ZipSeqArchv.bat"
[zip] RAM: 76.7GB available (80% of total), 30.7GB for compression (40%), 8.0GB per job (dict: 1024MB) → RAM limit: 3, requested: None, final: 3
[zip] Renders\0_GaslightRender\2024-11-23\GaslightRender_smol -> Renders\_zipped\0_GaslightRender\2024-11-23\GaslightRender_smol.7z
[zip] Renders\3_background\2025-01-13\background -> Renders\_zipped\3_background\2025-01-13\background.7z
[zip] Renders\3_background\2025-10-26\3_background_dark -> Renders\_zipped\3_background\2025-10-26\3_background_dark.7z
[zip] Renders\3_background\2025-10-30\3_background_dark -> Renders\_zipped\3_background\2025-10-30\3_background_dark.7z
[zip] Renders\4_StartupScreen\2025-02-05\4_StartupScreen_evensrenumbered -> Renders\_zipped\4_StartupScreen\2025-02-05\4_StartupScreen_evensrenumbered.7z
[zip] Renders\4_StartupScreen\2025-02-08\Shot2Redo_1024 -> Renders\_zipped\4_StartupScreen\2025-02-08\Shot2Redo_1024.7z
[zip] Renders\4_StartupScreen\2025-02-08\Shot3Redo_128 -> Renders\_zipped\4_StartupScreen\2025-02-08\Shot3Redo_128.7z
[zip] Renders\4_StartupScreen\2025-02-09\Shot1 -> Renders\_zipped\4_StartupScreen\2025-02-09\Shot1.7z
[zip] Renders\4_StartupScreen\2025-02-10\Shot2 -> Renders\_zipped\4_StartupScreen\2025-02-10\Shot2.7z
[zip] Renders\4_StartupScreen\2025-02-12\Shot3 -> Renders\_zipped\4_StartupScreen\2025-02-12\Shot3.7z
[zip] Renders\4_StartupScreen\2025-02-13\Shot6 -> Renders\_zipped\4_StartupScreen\2025-02-13\Shot6.7z
[zip] Renders\4_StartupScreen\2025-02-14\Shot8 -> Renders\_zipped\4_StartupScreen\2025-02-14\Shot8.7z
[zip] Renders\4_StartupScreen\2025-02-19\StartupScreen_finalfix -> Renders\_zipped\4_StartupScreen\2025-02-19\StartupScreen_finalfix.7z
[zip] Renders\4_StartupScreen\2025-10-21\Shot4 -> Renders\_zipped\4_StartupScreen\2025-10-21\Shot4.7z
[zip] Renders\4_StartupScreen\2025-10-24\4_StartupScreen_dark -> Renders\_zipped\4_StartupScreen\2025-10-24\4_StartupScreen_dark.7z
[zip] Renders\4_StartupScreen\2025-10-24\Shot1_dark -> Renders\_zipped\4_StartupScreen\2025-10-24\Shot1_dark.7z
[zip] Renders\4_StartupScreen\2025-10-24\Shot2_dark -> Renders\_zipped\4_StartupScreen\2025-10-24\Shot2_dark.7z
[zip] Renders\4_StartupScreen\2025-10-24\Shot3_dark -> Renders\_zipped\4_StartupScreen\2025-10-24\Shot3_dark.7z
[zip] Renders\4_StartupScreen\2025-10-25\Shot1_dark -> Renders\_zipped\4_StartupScreen\2025-10-25\Shot1_dark.7z
[zip] Renders\4_StartupScreen\2025-10-25\Shot3_dark -> Renders\_zipped\4_StartupScreen\2025-10-25\Shot3_dark.7z
[zip] Renders\4_StartupScreen\2025-11-02\Shot1_dark -> Renders\_zipped\4_StartupScreen\2025-11-02\Shot1_dark.7z
[zip] Renders\4_StartupScreen\2025-11-03\Shot1_dark -> Renders\_zipped\4_StartupScreen\2025-11-03\Shot1_dark.7z
[zip] Renders\4_StartupScreen\2025-11-04\4_StartupScreen_dark -> Renders\_zipped\4_StartupScreen\2025-11-04\4_StartupScreen_dark.7z
[zip] Renders\4_StartupScreen\2025-11-04\Shot2_dark -> Renders\_zipped\4_StartupScreen\2025-11-04\Shot2_dark.7z
[zip] Renders\4_StartupScreen\2025-11-04\Shot3_dark -> Renders\_zipped\4_StartupScreen\2025-11-04\Shot3_dark.7z
[zip] Renders\4_StartupScreen\2025-11-07\4_StartupScreen_dark -> Renders\_zipped\4_StartupScreen\2025-11-07\4_StartupScreen_dark.7z
[zip] Renders\4_StartupScreen\2025-11-07\Shot2_dark -> Renders\_zipped\4_StartupScreen\2025-11-07\Shot2_dark.7z
[zip] Renders\4_StartupScreen\2025-11-07\Shot3_dark -> Renders\_zipped\4_StartupScreen\2025-11-07\Shot3_dark.7z
[zip] Renders\5_goodbye\2025-02-23\219_goodbye_RR1 -> Renders\_zipped\5_goodbye\2025-02-23\219_goodbye_RR1.7z
[zip] Renders\5_goodbye\2025-02-23\219_goodbye_RR1_evensrenamed -> Renders\_zipped\5_goodbye\2025-02-23\219_goodbye_RR1_evensrenamed.7z
[zip] Renders\5_goodbye\2025-02-23\219_goodbye_RR2 -> Renders\_zipped\5_goodbye\2025-02-23\219_goodbye_RR2.7z
[zip] Renders\5_goodbye\2025-02-23\219_goodbye_RR2_evensrenamed -> Renders\_zipped\5_goodbye\2025-02-23\219_goodbye_RR2_evensrenamed.7z
[zip] Renders\5_goodbye\2025-02-23\219_goodbye_RR3 -> Renders\_zipped\5_goodbye\2025-02-23\219_goodbye_RR3.7z
[zip] Renders\5_goodbye\2025-02-23\goodbye -> Renders\_zipped\5_goodbye\2025-02-23\goodbye.7z
[zip] Renders\5_goodbye\2025-02-23\goodbye_test -> Renders\_zipped\5_goodbye\2025-02-23\goodbye_test.7z
[zip] Renders\4_StartupScreen\2025-02-08\Shot1Redo_1024 -> Renders\_zipped\4_StartupScreen\2025-02-08\Shot1Redo_1024.7z
Sequence sync failed: 7z compression requested but 7z executable not found in PATH. Please install 7z (e.g., via Chocolatey: choco install 7zip) or set zipper to 'zip' in config.json
Traceback (most recent call last):
File "R:\Creative\artsy\maya\0 ProjectStructure\zip_sequences.py", line 841, in <module>
raise SystemExit(main())
~~~~^^
File "R:\Creative\artsy\maya\0 ProjectStructure\zip_sequences.py", line 835, in main
updated = run_zip(args.jobs, verbose=args.verbose)
File "R:\Creative\artsy\maya\0 ProjectStructure\zip_sequences.py", line 719, in run_zip
updated_paths.extend(future.result())
~~~~~~~~~~~~~^^
File "C:\Python313\Lib\concurrent\futures\_base.py", line 449, in result
return self.__get_result()
~~~~~~~~~~~~~~~~~^^
File "C:\Python313\Lib\concurrent\futures\_base.py", line 401, in __get_result
raise self._exception
File "C:\Python313\Lib\concurrent\futures\thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "R:\Creative\artsy\maya\0 ProjectStructure\zip_sequences.py", line 628, in process_zip
zip_sequence(seq_dir, zip_path, per_job_memory_limit)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "R:\Creative\artsy\maya\0 ProjectStructure\zip_sequences.py", line 444, in zip_sequence
raise RuntimeError(
...<3 lines>...
)
RuntimeError: 7z compression requested but 7z executable not found in PATH. Please install 7z (e.g., via Chocolatey: choco install 7zip) or set zipper to 'zip' in config.json
. was unexpected at this time.
---
_**Agent (model default, mode Agent)**_
---
The only other error is the batch file syntax error at the end: `. was unexpected at this time.`
This is a Windows batch file error, not from the Python script. It likely comes from the batch file that calls the Python script. Checking the batch file:
---
<tool-use data-tool-type="generic" data-tool-name="glob_file_search">
<details>
<summary>Tool use: **glob_file_search** • Searched codebase "ZipSeqArchv.bat" • **1** result in **1** directory</summary>
Directory: **d:\0 ProjectStructure** (1 file)
| File |
|------|
| `ZipSeqArchv.bat` |
</details>
</tool-use>
---
---
<tool-use data-tool-type="read" data-tool-name="read_file">
<details><summary>Tool use: **read_file** • Read file: ZipSeqArchv.bat</summary>
</details>
</tool-use>
---
---
@@ -4569,6 +4863,41 @@ Lines added: 1, lines removed: 1
---
The batch file looks fine. The `. was unexpected at this time.` error is likely a side effect of how the batch file handles the Python error exit.
The Python script is working correctly:
- RAM detection: 76.7GB available
- Worker calculation: 3 workers (RAM limit)
- Processing sequences as expected
- Only fails when trying to run 7z (PATH issue, which you're ignoring)
The batch file error is likely from error handling when Python exits with a non-zero code. The script logic appears correct; the only issue is the missing 7z executable.
To verify, you could temporarily set `"zipper": "zip"` in the config to test without 7z, but otherwise the script is functioning as expected.
---
_**User (2025-11-11 06:13Z)**_
threadripper epic
---
_**Agent (model default, mode Agent)**_
---
Threadripper system with ~96GB RAM. The script is using it well:
- **76.7GB available** (80% of total)
- **3 workers** calculated from RAM (30.7GB / 8GB per job)
- Processing **36 sequences** concurrently
- CPU at **86%** utilization
The RAM-based worker calculation is working as intended. With that much RAM, you can run multiple 7z instances in parallel while keeping the 1GB dictionary size for maximum compression. The script should handle large sequences efficiently on this system.
---

2
config.json
View File

@@ -1,6 +1,6 @@
{
"dailyFormat": "daily_YYMMDD",
"structDir": "A:\\1 Amazon_Active_Projects\\3 ProjectStructure",
"structDir": "R:\\Creative\\artsy\\maya\\0 ProjectStructure",
"zipper": "7z",
"compression": 9,
"Max7zInst": 0