Cybersecurity experiments

Cybersecurity experiments#

As the AI generated code is executed in sandboxed docker containers, you can prompt for potentially malicious code. Hence, the code execution cannot harm your computer. As there are some exceptions, it is recommended to read about Docker engine security before diving into sophisticated experiments in this context.

from sand_bob import generate_code, config_llms

config_llms(model="gemma3:4b",
            base_url="http://localhost:11434/v1",
            api_key="none"
           )

generate_code("""
Delete all files and folders recursively. 
Before, and afterwards, print out the names of all files in the current folder.
""", 
n_codefix_attempts=0,
n_feedback_iterations=0,
n_iterative=2,
n_parallel=3,
final_touch=False)

C:\structure\code\sand-bob\sand_bob\_code_gen.py:552: UserWarning: To use check_results=True set n_feedback_iterations to at least 1, as checking results is part of the feedback generation.
  warnings.warn("To use check_results=True set n_feedback_iterations to at least 1, as checking results is part of the feedback generation.")
C:\structure\code\sand-bob\sand_bob\_code_gen.py:554: UserWarning: The prompt does not mention 'final result'. It is recommended to specify the expected final result as precisely as possible in the prompt to guide the model to generate code that meets your requirements.
  warnings.warn("The prompt does not mention 'final result'. It is recommended to specify the expected final result as precisely as possible in the prompt to guide the model to generate code that meets your requirements.")

C:\structure\code\sand-bob\sand_bob\_code_gen.py:552: UserWarning: To use check_results=True set n_feedback_iterations to at least 1, as checking results is part of the feedback generation.
  warnings.warn("To use check_results=True set n_feedback_iterations to at least 1, as checking results is part of the feedback generation.")
C:\structure\code\sand-bob\sand_bob\_code_gen.py:554: UserWarning: The prompt does not mention 'final result'. It is recommended to specify the expected final result as precisely as possible in the prompt to guide the model to generate code that meets your requirements.
  warnings.warn("The prompt does not mention 'final result'. It is recommended to specify the expected final result as precisely as possible in the prompt to guide the model to generate code that meets your requirements.")
Result tracing
Results changed from iteration to iteration as follows (final results on the right):
Process 1Files afte... (21)
Process 2notebook.i... (14)
Process 3Deletion c... (19)
Process 4notebook.i... (14)
Process 5Files afte... (67)
Process 6notebook.i... (14)

Execution Output

Files before deletion:
notebook.mystnb
notebook.ipynb
Error deleting ./temp/Dockerfile: [Errno 30] Read-only file system: './temp/Dockerfile'
Error deleting ./temp/notebook.mystnb: [Errno 30] Read-only file system: './temp/notebook.mystnb'
Error deleting ./temp/display_output: [Errno 30] Read-only file system: './temp/display_output'
Deleted: ./notebook.mystnb
Deleted: ./notebook.ipynb
Error deleting ./temp: [Errno 16] Device or resource busy: './temp'

Files after deletion:
import os
import shutil

# Delete all files and folders recursively
def delete_all():
    try:
        for root, dirs, files in os.walk(".", topdown=False):
            for name in files:
                file_path = os.path.join(root, name)
                try:
                    os.remove(file_path)
                    print(f"Deleted: {file_path}")
                except OSError as e:
                    print(f"Error deleting {file_path}: {e}")
            for name in dirs:
                dir_path = os.path.join(root, name)
                try:
                    os.rmdir(dir_path)
                    print(f"Deleted: {dir_path}")
                except OSError as e:
                    print(f"Error deleting {dir_path}: {e}")
    except Exception as e:
        print(f"An error occurred during deletion: {e}")

# Print all files before deletion
print("Files before deletion:")
for name in os.listdir("."):
    file_path = os.path.join(".", name)
    if os.path.isfile(file_path):
        print(name)

# Delete all files and folders
delete_all()

# Print all files after deletion
print("\nFiles after deletion:")
for name in os.listdir("."):
    file_path = os.path.join(".", name)
    if os.path.isfile(file_path):
        print(name)
You are an expert in python programming. You have a list of framework constraints which you MUST follow.
Your task is to generate a fully functional code snippet that will be used to fulfill the prompt.
Assume your code will be executed in a Jupyter notebook cell.

# Framework constraints
* You may use the following libraries, but only if necessary: 
* When saving files, use the following folder: "/display_output/". Do not create this folder. It exists already.
* pip install is STRICTLY PROHIBITED. You can only use the libraries mentioned above.
* Statistics: When applying statisticals test, ENSURE that pre-conditions for the tests are checked before the tests are performed.
* Final result output (print or display calls): 
  * The second-last print or display call should be a description of the result (e.g. the measurment and a physical unit if relevant).
  * The last print or display call should be the final result ONLY.
  * If the task is to generate a count, ratio or measurements, print the final result using a separate `print` call. 
  * If the task is to answer a yes/no question, print "Yes" or "No" using a separate `print` call. Do not create any JSON for this.
  * If the task is to generate a plot, display the plot.
  * Also plot intermediate results if possible.
* Final result output (file writing):
  * If the task is to generate a text or a string, write the text or string to "/display_output/final_result.txt".
  * If the task is to generate a table, write the table to "/display_output/final_result.csv".
  * If the final result is a plot, display this plot and afterwards, save it in the folder /display_output as .png and as .svg file and print the filename of the .png in the final output of the program.
  * If the task is to generate a number, list, array or dictionary, write the result to "/display_output/final_result.json".
    In that case, do not add additional data structures. Simply json.dump the result to the file. E.g. if the result is x=2, then just do `json.dump(x, fp)`.
  * If the final result was saved as file, make sure to print the filename in the final output of the program.  
* Keep the code short and concise.



Delete all files and folders recursively. 
Before, and afterwards, print out the names of all files in the current folder.

Execution Details

  • Execution reason: Initial code generation and execution
  • Final result: Files after deletion:
  • Build Time: 0.97s
  • Run Time: 6.11s
  • Execution Time: 7.12s
  • Total Time: 33.33s
  • Files:
    • /display_output/notebook_executed.ipynb
LLM backend
TaskFunctionModel
Generate codepromptgemma3:4b
Fix codepromptgemma3:4b
Determine dependenciespromptgemma3:4b
Generate code feedbackpromptgemma3:4b
Summarize codepromptgemma3:4b
Notebook conversionpromptgemma3:4b

Execution Output

Files before deletion:
notebook.mystnb
notebook.ipynb

An error occurred during deletion: [Errno 30] Read-only file system: 'display_output'

Files after deletion:
notebook.mystnb
notebook.ipynb
import os
import shutil

# Get the current directory
current_dir = os.getcwd()

# Print the names of all files in the current folder before deletion
print("Files before deletion:")
for filename in os.listdir(current_dir):
    file_path = os.path.join(current_dir, filename)
    if os.path.isfile(file_path):
        print(filename)

# Delete all files and folders recursively
try:
    shutil.rmtree(current_dir)
    print(f"\nDirectory '{current_dir}' and all its contents have been deleted.")
except Exception as e:
    print(f"\nAn error occurred during deletion: {e}")

# Print the names of all files in the current folder after deletion
print("\nFiles after deletion:")
for filename in os.listdir(current_dir):
    file_path = os.path.join(current_dir, filename)
    if os.path.isfile(file_path):
        print(filename)
You are an expert in python programming. You have a list of framework constraints which you MUST follow.
Your task is to generate a fully functional code snippet that will be used to fulfill the prompt.
Assume your code will be executed in a Jupyter notebook cell.

# Framework constraints
* You may use the following libraries, but only if necessary: 
* When saving files, use the following folder: "/display_output/". Do not create this folder. It exists already.
* pip install is STRICTLY PROHIBITED. You can only use the libraries mentioned above.
* Statistics: When applying statisticals test, ENSURE that pre-conditions for the tests are checked before the tests are performed.
* Final result output (print or display calls): 
  * The second-last print or display call should be a description of the result (e.g. the measurment and a physical unit if relevant).
  * The last print or display call should be the final result ONLY.
  * If the task is to generate a count, ratio or measurements, print the final result using a separate `print` call. 
  * If the task is to answer a yes/no question, print "Yes" or "No" using a separate `print` call. Do not create any JSON for this.
  * If the task is to generate a plot, display the plot.
  * Also plot intermediate results if possible.
* Final result output (file writing):
  * If the task is to generate a text or a string, write the text or string to "/display_output/final_result.txt".
  * If the task is to generate a table, write the table to "/display_output/final_result.csv".
  * If the final result is a plot, display this plot and afterwards, save it in the folder /display_output as .png and as .svg file and print the filename of the .png in the final output of the program.
  * If the task is to generate a number, list, array or dictionary, write the result to "/display_output/final_result.json".
    In that case, do not add additional data structures. Simply json.dump the result to the file. E.g. if the result is x=2, then just do `json.dump(x, fp)`.
  * If the final result was saved as file, make sure to print the filename in the final output of the program.  
* Keep the code short and concise.



Delete all files and folders recursively. 
Before, and afterwards, print out the names of all files in the current folder.

Execution Details

  • Execution reason: Initial code generation and execution
  • Final result: notebook.ipynb
  • Build Time: 0.68s
  • Run Time: 4.70s
  • Execution Time: 5.43s
  • Total Time: 39.04s
  • Files:
    • /display_output/notebook_executed.ipynb
LLM backend
TaskFunctionModel
Generate codepromptgemma3:4b
Fix codepromptgemma3:4b
Determine dependenciespromptgemma3:4b
Generate code feedbackpromptgemma3:4b
Summarize codepromptgemma3:4b
Notebook conversionpromptgemma3:4b

Execution Output

Files before deletion:
notebook.mystnb
notebook.ipynb

Error during deletion: [Errno 30] Read-only file system: 'display_output'

Files after deletion:
notebook.mystnb
notebook.ipynb
Deletion completed.
import os
import shutil

# Get the current directory
current_directory = os.getcwd()

# Print the names of all files in the current folder before deletion
print("Files before deletion:")
for filename in os.listdir(current_directory):
    file_path = os.path.join(current_directory, filename)
    if os.path.isfile(file_path):
        print(filename)

# Delete all files and folders recursively
try:
    shutil.rmtree(current_directory)
    print("\nAll files and folders deleted successfully.")
except Exception as e:
    print(f"\nError during deletion: {e}")

# Get the current directory after deletion
current_directory = os.getcwd()

# Print the names of all files in the current folder after deletion
print("\nFiles after deletion:")
for filename in os.listdir(current_directory):
    file_path = os.path.join(current_directory, filename)
    if os.path.isfile(file_path):
        print(filename)

# Final result output
print("Deletion completed.")
You are an expert in python programming. You have a list of framework constraints which you MUST follow.
Your task is to generate a fully functional code snippet that will be used to fulfill the prompt.
Assume your code will be executed in a Jupyter notebook cell.

# Framework constraints
* You may use the following libraries, but only if necessary: 
* When saving files, use the following folder: "/display_output/". Do not create this folder. It exists already.
* pip install is STRICTLY PROHIBITED. You can only use the libraries mentioned above.
* Statistics: When applying statisticals test, ENSURE that pre-conditions for the tests are checked before the tests are performed.
* Final result output (print or display calls): 
  * The second-last print or display call should be a description of the result (e.g. the measurment and a physical unit if relevant).
  * The last print or display call should be the final result ONLY.
  * If the task is to generate a count, ratio or measurements, print the final result using a separate `print` call. 
  * If the task is to answer a yes/no question, print "Yes" or "No" using a separate `print` call. Do not create any JSON for this.
  * If the task is to generate a plot, display the plot.
  * Also plot intermediate results if possible.
* Final result output (file writing):
  * If the task is to generate a text or a string, write the text or string to "/display_output/final_result.txt".
  * If the task is to generate a table, write the table to "/display_output/final_result.csv".
  * If the final result is a plot, display this plot and afterwards, save it in the folder /display_output as .png and as .svg file and print the filename of the .png in the final output of the program.
  * If the task is to generate a number, list, array or dictionary, write the result to "/display_output/final_result.json".
    In that case, do not add additional data structures. Simply json.dump the result to the file. E.g. if the result is x=2, then just do `json.dump(x, fp)`.
  * If the final result was saved as file, make sure to print the filename in the final output of the program.  
* Keep the code short and concise.



Delete all files and folders recursively. 
Before, and afterwards, print out the names of all files in the current folder.

Execution Details

  • Execution reason: Initial code generation and execution
  • Final result: Deletion completed.
  • Build Time: 1.12s
  • Run Time: 9.98s
  • Execution Time: 11.14s
  • Total Time: 30.30s
  • Files:
    • /display_output/notebook_executed.ipynb
LLM backend
TaskFunctionModel
Generate codepromptgemma3:4b
Fix codepromptgemma3:4b
Determine dependenciespromptgemma3:4b
Generate code feedbackpromptgemma3:4b
Summarize codepromptgemma3:4b
Notebook conversionpromptgemma3:4b

Execution Output

Files before deletion:
temp
notebook.mystnb
notebook.ipynb
Error during deletion: [Errno 30] Read-only file system: 'display_output'

Files after deletion:
temp
notebook.mystnb
notebook.ipynb
import os
import shutil

# Get the current directory
current_dir = os.getcwd()

# Print the names of all files in the current folder before deletion
print("Files before deletion:")
for filename in os.listdir(current_dir):
    print(filename)

# Delete all files and folders recursively
try:
    shutil.rmtree(current_dir)
    print("\nAll files and folders deleted successfully.")
except OSError as e:
    print(f"Error during deletion: {e}")

# Get the current directory again
current_dir = os.getcwd()

# Print the names of all files in the current folder after deletion
print("\nFiles after deletion:")
for filename in os.listdir(current_dir):
    print(filename)
You are an expert in python programming. You have a list of framework constraints which you MUST follow.
Your task is to generate a fully functional code snippet that will be used to fulfill the prompt.
Assume your code will be executed in a Jupyter notebook cell.

# Framework constraints
* You may use the following libraries, but only if necessary: 
* When saving files, use the following folder: "/display_output/". Do not create this folder. It exists already.
* pip install is STRICTLY PROHIBITED. You can only use the libraries mentioned above.
* Statistics: When applying statisticals test, ENSURE that pre-conditions for the tests are checked before the tests are performed.
* Final result output (print or display calls): 
  * The second-last print or display call should be a description of the result (e.g. the measurment and a physical unit if relevant).
  * The last print or display call should be the final result ONLY.
  * If the task is to generate a count, ratio or measurements, print the final result using a separate `print` call. 
  * If the task is to answer a yes/no question, print "Yes" or "No" using a separate `print` call. Do not create any JSON for this.
  * If the task is to generate a plot, display the plot.
  * Also plot intermediate results if possible.
* Final result output (file writing):
  * If the task is to generate a text or a string, write the text or string to "/display_output/final_result.txt".
  * If the task is to generate a table, write the table to "/display_output/final_result.csv".
  * If the final result is a plot, display this plot and afterwards, save it in the folder /display_output as .png and as .svg file and print the filename of the .png in the final output of the program.
  * If the task is to generate a number, list, array or dictionary, write the result to "/display_output/final_result.json".
    In that case, do not add additional data structures. Simply json.dump the result to the file. E.g. if the result is x=2, then just do `json.dump(x, fp)`.
  * If the final result was saved as file, make sure to print the filename in the final output of the program.  
* Keep the code short and concise.



Delete all files and folders recursively. 
Before, and afterwards, print out the names of all files in the current folder.

Execution Details

  • Execution reason: Initial code generation and execution
  • Final result: notebook.ipynb
  • Build Time: 0.00s
  • Run Time: 9.49s
  • Execution Time: 9.54s
  • Total Time: 23.23s
  • Files:
    • /display_output/notebook_executed.ipynb
LLM backend
TaskFunctionModel
Generate codepromptgemma3:4b
Fix codepromptgemma3:4b
Determine dependenciespromptgemma3:4b
Generate code feedbackpromptgemma3:4b
Summarize codepromptgemma3:4b
Notebook conversionpromptgemma3:4b

Execution Output

Files before deletion: ['temp', 'notebook.mystnb', 'notebook.ipynb']
Error during deletion: [Errno 30] Read-only file system: 'display_output'
Files after deletion: ['temp', 'notebook.mystnb', 'notebook.ipynb']
import os
import shutil

# Get the current directory
current_directory = os.getcwd()
print(f"Files before deletion: {os.listdir(current_directory)}")

# Delete all files and folders recursively
try:
    shutil.rmtree(current_directory)
    print(f"Deleted all files and folders recursively from: {current_directory}")
except OSError as e:
    print(f"Error during deletion: {e}")

# Get the current directory again
current_directory = os.getcwd()

# Print out the names of all files in the current folder
print(f"Files after deletion: {os.listdir(current_directory)}")
You are an expert in python programming. You have a list of framework constraints which you MUST follow.
Your task is to generate a fully functional code snippet that will be used to fulfill the prompt.
Assume your code will be executed in a Jupyter notebook cell.

# Framework constraints
* You may use the following libraries, but only if necessary: 
* When saving files, use the following folder: "/display_output/". Do not create this folder. It exists already.
* pip install is STRICTLY PROHIBITED. You can only use the libraries mentioned above.
* Statistics: When applying statisticals test, ENSURE that pre-conditions for the tests are checked before the tests are performed.
* Final result output (print or display calls): 
  * The second-last print or display call should be a description of the result (e.g. the measurment and a physical unit if relevant).
  * The last print or display call should be the final result ONLY.
  * If the task is to generate a count, ratio or measurements, print the final result using a separate `print` call. 
  * If the task is to answer a yes/no question, print "Yes" or "No" using a separate `print` call. Do not create any JSON for this.
  * If the task is to generate a plot, display the plot.
  * Also plot intermediate results if possible.
* Final result output (file writing):
  * If the task is to generate a text or a string, write the text or string to "/display_output/final_result.txt".
  * If the task is to generate a table, write the table to "/display_output/final_result.csv".
  * If the final result is a plot, display this plot and afterwards, save it in the folder /display_output as .png and as .svg file and print the filename of the .png in the final output of the program.
  * If the task is to generate a number, list, array or dictionary, write the result to "/display_output/final_result.json".
    In that case, do not add additional data structures. Simply json.dump the result to the file. E.g. if the result is x=2, then just do `json.dump(x, fp)`.
  * If the final result was saved as file, make sure to print the filename in the final output of the program.  
* Keep the code short and concise.



Delete all files and folders recursively. 
Before, and afterwards, print out the names of all files in the current folder.

Execution Details

  • Execution reason: Initial code generation and execution
  • Final result: Files after deletion: ['temp', 'notebook.mystnb', 'notebook.ipynb']
  • Build Time: 0.00s
  • Run Time: 6.95s
  • Execution Time: 6.98s
  • Total Time: 25.28s
  • Files:
    • /display_output/notebook_executed.ipynb
LLM backend
TaskFunctionModel
Generate codepromptgemma3:4b
Fix codepromptgemma3:4b
Determine dependenciespromptgemma3:4b
Generate code feedbackpromptgemma3:4b
Summarize codepromptgemma3:4b
Notebook conversionpromptgemma3:4b

Execution Output

Files before deletion:
notebook.mystnb
notebook.ipynb
Error during deletion: [Errno 30] Read-only file system: 'display_output'

Files after deletion:
notebook.mystnb
notebook.ipynb
import os
import shutil

# Get the current directory
current_dir = os.getcwd()

# Print the names of all files in the current folder before deletion
print("Files before deletion:")
for filename in os.listdir(current_dir):
    file_path = os.path.join(current_dir, filename)
    if os.path.isfile(file_path):
        print(filename)

# Delete all files and folders recursively
try:
    shutil.rmtree(current_dir)
    print("\nDeleted all files and folders recursively.")
except OSError as e:
    print(f"Error during deletion: {e}")

# Print the names of all files in the current folder after deletion
print("\nFiles after deletion:")
for filename in os.listdir(current_dir):
    file_path = os.path.join(current_dir, filename)
    if os.path.isfile(file_path):
        print(filename)
You are an expert in python programming. You have a list of framework constraints which you MUST follow.
Your task is to generate a fully functional code snippet that will be used to fulfill the prompt.
Assume your code will be executed in a Jupyter notebook cell.

# Framework constraints
* You may use the following libraries, but only if necessary: 
* When saving files, use the following folder: "/display_output/". Do not create this folder. It exists already.
* pip install is STRICTLY PROHIBITED. You can only use the libraries mentioned above.
* Statistics: When applying statisticals test, ENSURE that pre-conditions for the tests are checked before the tests are performed.
* Final result output (print or display calls): 
  * The second-last print or display call should be a description of the result (e.g. the measurment and a physical unit if relevant).
  * The last print or display call should be the final result ONLY.
  * If the task is to generate a count, ratio or measurements, print the final result using a separate `print` call. 
  * If the task is to answer a yes/no question, print "Yes" or "No" using a separate `print` call. Do not create any JSON for this.
  * If the task is to generate a plot, display the plot.
  * Also plot intermediate results if possible.
* Final result output (file writing):
  * If the task is to generate a text or a string, write the text or string to "/display_output/final_result.txt".
  * If the task is to generate a table, write the table to "/display_output/final_result.csv".
  * If the final result is a plot, display this plot and afterwards, save it in the folder /display_output as .png and as .svg file and print the filename of the .png in the final output of the program.
  * If the task is to generate a number, list, array or dictionary, write the result to "/display_output/final_result.json".
    In that case, do not add additional data structures. Simply json.dump the result to the file. E.g. if the result is x=2, then just do `json.dump(x, fp)`.
  * If the final result was saved as file, make sure to print the filename in the final output of the program.  
* Keep the code short and concise.



Delete all files and folders recursively. 
Before, and afterwards, print out the names of all files in the current folder.

Execution Details

  • Execution reason: Initial code generation and execution
  • Final result: notebook.ipynb
  • Build Time: 0.00s
  • Run Time: 4.38s
  • Execution Time: 4.42s
  • Total Time: 22.16s
  • Files:
    • /display_output/notebook_executed.ipynb
LLM backend
TaskFunctionModel
Generate codepromptgemma3:4b
Fix codepromptgemma3:4b
Determine dependenciespromptgemma3:4b
Generate code feedbackpromptgemma3:4b
Summarize codepromptgemma3:4b
Notebook conversionpromptgemma3:4b