ResultsCollector¶
The ResultsCollector handles collection and organization of results from temp files generated by the experiments.util.estimate script.
Utility class to collect the results of estimation and make a dataframe from all the models and cases.
- class experiments.util.collect_results.ResultsCollector(batch_size: int, input_channels: int, input_HW: int, num_classes: int, device: str, architecture: str, vjp_improvements: ~typing.List[float], cases: ~typing.List[~typing.List[str] | None], results_dir: str, print: ~typing.Callable = <bound method tqdm.write of <class 'tqdm.std.tqdm'>>)¶
This class collects results by reading from the results/ directory
Initialize the collector before all runs.
- Parameters:
batch_size (int) – batch_size
input_channels (int) – input_channels
input_HW (int) – input_HW
num_classes (int) – num_classes
device (str) – device
architecture (str) – conv or linear
vjp_improvements (List[float]) – vjp_improvements
cases (List[Union[None, List[str]]]) – list of cases
results_dir (str) – The base results dir
print (Callable, optional) – Which function to use for printing (i.e. print() causes problems in a tqdm context)
- clear_file(estimate: str)¶
Clears the temp file for the given estimation, to be called before and after collecting stats for a run.
- Parameters:
estimate (str) – time or memory
- collect_from_file(estimate: str, model: str)¶
To be called after all cases of a model have finished.
- Parameters:
- Raises:
AssertionError – If the temp file has more/less lines than expected (= number of cases)
ValueError – If the temp file has unallowed text (only floats are allowed)
- finish()¶
To be called after ALL cases on all models have been run, saves dataframes to csv files.
- experiments.util.collect_results.hyperparam_str(args: SimpleNamespace) str¶
Format hyperparams into a string
- Parameters:
args – args
- Returns:
Output
- Return type:
- experiments.util.collect_results.make_case_str(case: None | List[str]) str¶
Format case into a string