[{"content":" Period2022 – 2023 WhereSapienza University of Rome · EPFL VITA Lab RoleMSc thesis StatusCompleted Code \u0026amp; thesis on GitHub ↗ MSc thesis, A Deep Learning Architecture for Enhanced Depth Estimation on an Unrectified Stereo 360° Dataset, Sapienza University of Rome. Supervisors: Prof. Antonello Rizzi (Sapienza), Prof. Alexandre Alahi (EPFL).\nThe problem # Stereo depth networks are trained almost entirely on synthetic data and on rectified pinhole image pairs. Real 360° cameras on a vehicle give neither: the images are heavily distorted equirectangular panoramas, and rectifying them needs careful calibration that does not survive outside the lab. Existing 360° depth datasets (Matterport3D, Stanford 3D) are synthetic or rendered, so there was nothing realistic to train or evaluate on.\nWhat I built # A new dataset. I designed and recorded a real-world dataset with two RICOH Theta 360° cameras stacked vertically above an Ouster OS1-64 LiDAR, at EPFL in Lausanne.\n17,373 time-synchronised tuples (upper image, lower image, LiDAR depth image and point cloud) across two outdoor street scenes and two indoor corridors. 3840 × 1920 stereo images, with LiDAR depth images at 1024 × 64 as ground truth (depth, not disparity). Deliberately left unrectified, to make the dataset closer to what a deployed system sees. I removed the LiDAR distortion and aligned its panoramic output with the camera format so the three streams line up. A network that works on it. I benchmarked PSMNet and 360SD-Net, then modified 360SD-Net to train on the new data: a feature extractor with residual blocks and dilated convolutions, an ASPP module for context, a cost volume regularised by a stacked-hourglass 3D CNN, and depth regression. The polar angle of each pixel is fed in through a separate branch and fused late, to give the network the distortion pattern of the equirectangular projection. Trained for 500 epochs in PyTorch.\nResults # Out of the box, 360SD-Net did not cope with the unrectified pairs (RMSE in the thousands on the Outdoor 1 scene); the modified network reached an RMSE of 0.9 on the same scene. Ablations showed larger input crops help (256 × 512 beat 256 × 256), while the polar-angle branch contributed little on this dataset, so it can be dropped to cut compute. The dataset, released to the lab, gives a harder and more realistic benchmark for 360° stereo depth than the synthetic alternatives. ","date":"March 2023","externalUrl":null,"permalink":"/projects/360-stereo-depth-estimation/","section":"Projects","summary":"MSc thesis: recorded a new real-world dataset of 17,373 stereo 360° image pairs with LiDAR depth ground truth, and adapted deep stereo networks (PSMNet, 360SD-Net) to work on it without rectification.","title":"Stereo Depth Estimation on a Real, Unrectified 360° Dataset","type":"projects"},{"content":" PeriodSpring 2022 WhereEPFL, Deep Learning for Autonomous Vehicles (DLAV) RoleCourse project StatusCompleted Code on GitHub ↗ The problem # In the DLAV final race, a Segway Loomo robot has to follow one specific person through a course that other people are walking through. The robot must decide who to follow without any manual labelling, and then not lose or swap that person when others cross in front.\nHow it works # The pipeline has two stages, switched by a small state machine.\nFind the person of interest. Every frame goes through AlphaPose (ResNet-50 backbone, YOLO person detector). The person of interest is whoever makes the start gesture, one hand raised above the shoulder and the other below it, checked from the shoulder and wrist keypoints with a confidence threshold so half-occluded people are skipped. Track them. Once found, the pipeline hands over to a YOLOv5 detector with DeepSORT and an OSNet re-identification model. The person of interest is matched to a DeepSORT track ID by the distance between box centres (an IoU-based matcher is also implemented), and from then on the robot follows that ID, so it holds on through crossings and short occlusions. The output to the robot\u0026rsquo;s controller is the bounding box of the person to follow. All inference ran on the Loomo\u0026rsquo;s onboard computer, so the robot followed the chosen person in real time with no offboard compute.\nStack # Python, PyTorch, AlphaPose, YOLOv5, DeepSORT with OSNet re-ID, OpenCV. An earlier variant with YOLOv4 (TensorFlow) and DeepSORT is also in the repo.\n","date":"June 2022","externalUrl":null,"permalink":"/projects/loomo-person-tracking/","section":"Projects","summary":"Real-time perception pipeline for EPFL’s human–robot tandem race: a Segway Loomo picks out the person who raises a hand using AlphaPose keypoints, then keeps them locked in with YOLOv5 + DeepSORT while it follows them.","title":"Gesture-Triggered Person Following on a Loomo Robot","type":"projects"},{"content":" PeriodMSc coursework WhereSapienza University of Rome, Computational Intelligence (Prof. Antonello Rizzi) RoleCo-author, with Peyman Javaheri StatusCompleted Code \u0026amp; notebooks on GitHub ↗ The problem # The Game of Deep Learning ship dataset comes with class labels only. The goal here was both to say what kind of ship is in an image and where it is, so the localisation labels had to be created first.\nWhat we did # Annotation. Drew bounding boxes for 6,182 images with LabelImg (2,125 cargo, 916 carrier, 776 cruise, 1,148 military, 1,217 tanker), then converted the XML annotations to CSV for training. The annotated CSVs are in the repo. Model 1, multi-output. One MobileNetV2 backbone with two heads: a softmax classifier over the five classes and a regression head for the bounding box. Trained jointly from a custom generator yielding images, labels and boxes. Model 2, split. A dedicated Xception classifier plus a separate MobileNetV2 box regressor, each trained on its own task with augmentation (flips, rotations). Evaluation. Classification accuracy for the label, IoU between predicted and true boxes for localisation, and a demo notebook that loads the trained weights and draws predictions and box centroids on test images. On the nine demo images shown above, every ship is assigned the correct class and box IoU ranges from 56% to 88%. Stack # Python, TensorFlow/Keras, MobileNetV2, Xception, LabelImg, Google Colab.\n","date":"June 2022","externalUrl":null,"permalink":"/projects/ship-classification-localisation/","section":"Projects","summary":"Classify ships into five types (cargo, carrier, cruise, military, tanker) and localise them in the image, on ~6,200 hand-annotated images with bounding boxes. Compares a single multi-output MobileNetV2 against a split Xception + MobileNetV2 pair.","title":"Ship Classification and Localisation","type":"projects"},{"content":"I build deep reinforcement learning systems for sequential decision making under uncertainty: multi-objective, constrained and multi-agent policies for battery dispatch, energy management and fair resource sharing. Before that I worked on 360° stereo depth estimation and real-time person tracking for robots at EPFL.\nAvailable from November 2026.\nCV Publications Projects ","date":"March 2026","externalUrl":null,"permalink":"/","section":"Danial Zendehdel","summary":"","title":"Danial Zendehdel","type":"page"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"Devops","type":"tags"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/digitalocean/","section":"Tags","summary":"","title":"Digitalocean","type":"tags"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/dns/","section":"Tags","summary":"","title":"Dns","type":"tags"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/github/","section":"Tags","summary":"","title":"Github","type":"tags"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/namecheap/","section":"Tags","summary":"","title":"Namecheap","type":"tags"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/perks/","section":"Perks","summary":"","title":"Perks","type":"perks"},{"content":" Setting Up a Custom Domain for GitHub Pages # This is a real-world walkthrough — not a clean tutorial. Every problem listed here actually happened.\nThe Goal # Take a Hugo site deployed on GitHub Pages at:\nhttps://username.github.io/my-repo/ And serve it at a clean custom domain:\nhttps://danialzendehdel.com Step 1 — Add DNS Records # GitHub Pages requires 4 A records pointing to their servers, plus a CNAME for www.\nGitHub Pages IPs (A Records) # 185.199.108.153 185.199.109.153 185.199.110.153 185.199.111.153 DNS Records to Add # Type Host Value TTL A @ 185.199.108.153 300 A @ 185.199.109.153 300 A @ 185.199.110.153 300 A @ 185.199.111.153 300 CNAME www yourdomain.com. 300 TTL note: Use 300 (5 minutes) while testing — faster propagation. Once confirmed working, bump to 3600 (1 hour) for stability.\nStep 2 — Add CNAME File to Your Repo # GitHub Pages needs to know your custom domain. Create static/CNAME in your Hugo project:\ndanialzendehdel.com Hugo will copy this to the root of the built site automatically.\nStep 3 — Update hugo.toml baseURL # baseURL = \u0026#34;https://danialzendehdel.com/\u0026#34; This is critical. Without it, GitHub Actions will build the site with the wrong base URL and all CSS/JS paths will be broken.\nStep 4 — Configure GitHub Pages # GitHub → repo → Settings → Pages → Custom domain\nType your domain, click Save. Wait for the DNS check to go green ✅, then enable Enforce HTTPS.\n⚠️ This step triggers a rebuild. If you skip it, the GitHub Actions workflow will still output the old base URL — even if hugo.toml is correct.\nProblems We Hit (And How to Fix Them) # Problem 1 — NXDOMAIN (Domain Not Found) # Querying dns.google returned:\n{ \u0026#34;Status\u0026#34;: 3 } Status 3 = NXDOMAIN = domain doesn\u0026rsquo;t exist in global DNS.\nCause: The domain\u0026rsquo;s nameservers at the registrar weren\u0026rsquo;t pointing to DigitalOcean.\nHow to check:\ncurl -s \u0026#34;https://dns.google/resolve?name=yourdomain.com\u0026amp;type=NS\u0026#34; If the Answer array shows dns1.registrar-servers.com → you\u0026rsquo;re using Namecheap\u0026rsquo;s nameservers, not DigitalOcean\u0026rsquo;s.\nProblem 2 — DNS Records in DigitalOcean Being Ignored # You added A records in DigitalOcean\u0026rsquo;s Networking → Domains panel, but the domain resolves to a different IP.\nCause: DigitalOcean\u0026rsquo;s DNS panel ≠ your domain\u0026rsquo;s authoritative nameserver. Just adding a domain to DO\u0026rsquo;s panel doesn\u0026rsquo;t make DO authoritative for it — you need to update the nameservers at your registrar.\nTwo fixes:\nOption A — Keep Namecheap DNS, just update records there: Log into Namecheap → Advanced DNS → add the 4 A records + CNAME directly in Namecheap.\nOption B — Switch to DigitalOcean DNS: Namecheap → Nameservers → Custom DNS:\nns1.digitalocean.com ns2.digitalocean.com ns3.digitalocean.com Then your DO panel records take effect.\nProblem 3 — URL Redirect Overriding A Records # Even after adding correct A records in Namecheap, the domain still resolved to the wrong IP.\nCause: Namecheap had a URL Redirect Record (@ → http://www.yourdomain.com) left over from the default parking setup. This intercepts traffic before A records are checked.\nFix: Delete:\nURL Redirect Record for @ Old CNAME for www → parkingpage.namecheap.com Keep only:\n4 A records pointing to GitHub Pages CNAME www → yourdomain.com. Problem 4 — CSS/JS Not Loading After Domain Switch # Site loads but renders blank white or completely unstyled.\nCause: The HTML contains:\n\u0026lt;base href=\u0026#34;https://username.github.io/my-repo/\u0026#34;\u0026gt; All asset paths (CSS, JS) are relative to this old base URL, so they 404 on the new domain.\nRoot cause: GitHub Actions uses ${{ steps.pages.outputs.base_url }} for the base URL. This only updates to your custom domain after you set it in GitHub Pages Settings. The baseURL in hugo.toml is overridden by the workflow flag --baseURL \u0026quot;${{ steps.pages.outputs.base_url }}/\u0026quot;.\nFix: Set the custom domain in GitHub Pages Settings → this triggers a correct rebuild.\nProblem 5 — Layout Broken (Mobile Stacked View) # The site loaded but the sidebar was stacked on top of the content instead of appearing on the left.\nCause: The Poison/Hyde theme uses body { overflow: hidden; height: 100vh; } with a flex row layout. On viewports narrower than 48em (768px), it switches to a stacked column layout.\nThis isn\u0026rsquo;t a bug — it\u0026rsquo;s responsive design. The site looks correct at full desktop width.\nDNS Propagation Times # Change Type Typical Time A record update 5 min – 1 hour CNAME update 5 min – 1 hour Nameserver change 1 – 24 hours New domain purchase 1 – 24 hours How to check propagation:\ncurl -s \u0026#34;https://dns.google/resolve?name=yourdomain.com\u0026amp;type=A\u0026#34; If Answer array has IPs → propagated ✅\nIf empty or Status 3 → still waiting ⏳\nFinal Checklist # 4 A records added at your actual registrar (not just DO panel) CNAME www added Old parking redirects deleted static/CNAME file in Hugo repo baseURL updated in hugo.toml Custom domain set in GitHub Pages Settings DNS check passed ✅ in GitHub Settings HTTPS enforced ✅ Quick DNS Debug Commands # # Check A records curl -s \u0026#34;https://dns.google/resolve?name=yourdomain.com\u0026amp;type=A\u0026#34; # Check nameservers curl -s \u0026#34;https://dns.google/resolve?name=yourdomain.com\u0026amp;type=NS\u0026#34; # Check CNAME curl -s \u0026#34;https://dns.google/resolve?name=www.yourdomain.com\u0026amp;type=CNAME\u0026#34; # Check what\u0026#39;s actually serving curl -sI https://yourdomain.com | head -5 Painful to figure out the first time. Trivial once you know where to look.\n","date":"March 2026","externalUrl":null,"permalink":"/perks/custom-domain-dns/","section":"Perks","summary":"","title":"Setting Up a Custom Domain for GitHub Pages — The Full Journey","type":"perks"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/web/","section":"Tags","summary":"","title":"Web","type":"tags"},{"content":" How to Give Someone Access to a Single GitHub Repo # Need to let a machine, agent, or collaborator pull (or push) to one specific repo without granting access to your entire GitHub account? Deploy keys are the answer.\nA deploy key is an SSH key scoped to a single repository. Even if the private key is compromised, the attacker can only access that one repo — nothing else on your account.\nStep 1: Generate a Dedicated SSH Key # Run this on the machine that needs access:\nssh-keygen -t ed25519 -C \u0026#34;deploy-key-description\u0026#34; -f ~/.ssh/github_deploy_REPONAME -N \u0026#34;\u0026#34; -t ed25519 — modern, secure key type -N \u0026quot;\u0026quot; — no passphrase (needed for automated access) -f — saves to a specific file so it does not conflict with your main key Step 2: Add the Public Key to GitHub # Go to your repo on GitHub Navigate to Settings → Deploy keys → Add deploy key Paste the contents of ~/.ssh/github_deploy_REPONAME.pub Give it a name (e.g., my-server, ci-bot) Check \u0026ldquo;Allow write access\u0026rdquo; only if the machine needs to push Step 3: Configure SSH to Use This Key # Add an entry to ~/.ssh/config:\nHost github-REPONAME HostName github.com User git IdentityFile ~/.ssh/github_deploy_REPONAME IdentitiesOnly yes This creates a custom SSH alias so the deploy key is used only for this repo.\nStep 4: Clone or Update the Remote URL # New clone:\ngit clone git@github-REPONAME:yourusername/your-repo.git Existing repo — update the remote:\ngit remote set-url origin git@github-REPONAME:yourusername/your-repo.git Step 5: Test the Connection # ssh -T git@github-REPONAME You should see:\nHi yourusername/your-repo! You\u0026#39;ve successfully authenticated, but GitHub does not provide shell access. Why Deploy Keys Over Personal Access Tokens? # Method Scope Auth Best For Deploy Key Single repo SSH Servers, agents, CI PAT All repos HTTPS Scripts, APIs Deploy keys are safer for automated access — narrow scope, easy to revoke per-repo.\nQuick Summary # # 1. Generate key ssh-keygen -t ed25519 -C \u0026#34;deploy\u0026#34; -f ~/.ssh/github_deploy_REPO -N \u0026#34;\u0026#34; # 2. Add ~/.ssh/github_deploy_REPO.pub to GitHub repo Settings \u0026gt; Deploy keys # 3. Add to ~/.ssh/config Host github-REPO HostName github.com User git IdentityFile ~/.ssh/github_deploy_REPO IdentitiesOnly yes # 4. Clone git clone git@github-REPO:USERNAME/REPO.git Done. One key, one repo, zero blast radius.\n","date":"March 2026","externalUrl":null,"permalink":"/perks/repo-access/","section":"Perks","summary":"","title":"How to Give Someone Access to a Single GitHub Repo","type":"perks"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"","date":"March 2026","externalUrl":null,"permalink":"/tags/ssh/","section":"Tags","summary":"","title":"Ssh","type":"tags"},{"content":" Booleans in argparse (and how to fix it) # TL;DR: Don’t use type=bool for CLI flags. It turns strings like \u0026quot;False\u0026quot; into True. Use actions instead: action='store_true', action='store_false', or (Python ≥ 3.9) BooleanOptionalAction.\nWhy --verbose=False becomes True # argparse reads CLI values as strings. With type=bool, it effectively does:\nbool(\u0026#34;False\u0026#34;) # -\u0026gt; True (any non-empty string is truthy) So running:\npython app.py --verbose=False sets args.verbose to True — surprise!\nCorrect patterns # 1) One-way flags (present = True, absent = False) # parser.add_argument(\u0026#39;--verbose\u0026#39;, action=\u0026#39;store_true\u0026#39;, help=\u0026#39;enable verbose logging\u0026#39;) # default is False unless you pass --verbose If you want the inverse (default True, disable with a flag):\nparser.add_argument(\u0026#39;--no-cache\u0026#39;, action=\u0026#39;store_false\u0026#39;, dest=\u0026#39;cache\u0026#39;, help=\u0026#39;disable cache\u0026#39;) parser.set_defaults(cache=True) 2) Two-way flags (Python ≥ 3.9) # Gives you --feature and --no-feature.\nfrom argparse import BooleanOptionalAction parser.add_argument(\u0026#39;--verbose\u0026#39;, action=BooleanOptionalAction, default=False) # supports: --verbose / --no-verbose 3) If you must accept --flag=True|False # Use a small parser that understands common boolean spellings:\ndef str2bool(v): if isinstance(v, bool): return v v = v.lower() if v in (\u0026#39;1\u0026#39;,\u0026#39;true\u0026#39;,\u0026#39;t\u0026#39;,\u0026#39;yes\u0026#39;,\u0026#39;y\u0026#39;,\u0026#39;on\u0026#39;): return True if v in (\u0026#39;0\u0026#39;,\u0026#39;false\u0026#39;,\u0026#39;f\u0026#39;,\u0026#39;no\u0026#39;,\u0026#39;n\u0026#39;,\u0026#39;off\u0026#39;): return False raise argparse.ArgumentTypeError(\u0026#34;expected a boolean\u0026#34;) parser.add_argument(\u0026#39;--verbose\u0026#39;, type=str2bool, nargs=\u0026#39;?\u0026#39;, const=True, default=False) # --verbose -\u0026gt; True # --verbose true -\u0026gt; True # --verbose false -\u0026gt; False # (omitting it) -\u0026gt; False Drop-in snippet with fallback (works on all Python versions) # import argparse parser = argparse.ArgumentParser() try: # Python 3.9+ from argparse import BooleanOptionalAction parser.add_argument(\u0026#39;--verbose\u0026#39;, action=BooleanOptionalAction, default=False, help=\u0026#39;enable or disable verbose logs\u0026#39;) except Exception: # Older Pythons parser.add_argument(\u0026#39;--verbose\u0026#39;, action=\u0026#39;store_true\u0026#39;, help=\u0026#39;enable verbose logs\u0026#39;) # optional companion to explicitly disable (rarely needed): parser.add_argument(\u0026#39;--no-verbose\u0026#39;, dest=\u0026#39;verbose\u0026#39;, action=\u0026#39;store_false\u0026#39;) args = parser.parse_args() Bonus: converting to libraries that expect ints # Some libs (e.g., Stable-Baselines3) take verbose as an int. Just map:\nsb3_verbose = 1 if args.verbose else 0 Takeaway # Avoid type=bool in argparse. Prefer actions (store_true, store_false, BooleanOptionalAction). If you need --flag=True|False, supply a str2bool parser. Copy-paste this into your PR to save future-you (and reviewers) from confusing CLI behavior.\n","date":"August 2025","externalUrl":null,"permalink":"/issues/arg_parser/issue_argparser/","section":"ISSUES","summary":"","title":"argparser","type":"issues"},{"content":"","date":"August 2025","externalUrl":null,"permalink":"/issues/","section":"ISSUES","summary":"","title":"ISSUES","type":"issues"},{"content":" Gemini API # The required dependencies are:\npip install -U -q \u0026#34;google-generativeai\u0026gt;=0.8.3\u0026#34; As a first step, you need to create a Gemini account and get your API key. You can do this by visiting the Gemini API. Once you have your API key, you can use the Gemini class to interact.\nimport google.generativeai as genai import os genai.configure(api_key=os.environ[\u0026#34;GOOGLE_API_KEY\u0026#34;]) Note: Replace os.environ[\u0026quot;GOOGLE_API_KEY\u0026quot;] with your actual API key. You can create .env file in the root directory of your project and add the following line to it:\nGOOGLE_API_KEY=your_api_key First Prompt: # As a first step, you need to create a prompt. The prompt is a short description of the content you want to generate. For example, if you want to generate a sunset image, you can use the following prompt: \u0026ldquo;A beautiful sunset over the city skyline.\u0026rdquo; prompt = \u0026#34;A beautiful sunset over the city skyline.\u0026#34; model = genai.GenerativeModel(\u0026#34;gemini-1.5-flash\u0026#34;) response = model.generate_content(prompt) print(response.text) Start a chat # You can start a chat with the model by using the start_chat method. The model will respond to your messages based on the prompt you provide. While you have the chat object around, the conversation state persists. chat = model,start_chat(history=[]) response = chat.send_message(prompt) print(response.text) response = chat.send_message(\u0026#39;Can you tell something interesting about dinosaurs?\u0026#39;) print(response.text) Gemini models # The available models can be found models You can use the list_models method to get the list of available models. models = genai.list_models() print(models) Or print a model specificly to monitor the details of the model. model = genai.GenerativeModel(\u0026#34;gemini-1.5-flash\u0026#34;) print(model) Control Output length # When using a Large Language Model (LLM) to generate text, the length of the output has implications for cost and performance. Producing more tokens requires more computation, which in turn increases energy usage, response time, and overall cost.\nTo control the length of the output, you can set the max_output_tokens parameter in the Gemini API. This parameter limits the number of tokens the model will generate, but it does not affect the style or substance of the text produced. The text will simply stop once it reaches the set limit. To ensure the output is concise and complete within this constraint, careful crafting of the input prompt might be necessary.\nshort_model = genai.GenerativeModel( \u0026#39;gemini-1.5-flash\u0026#39;, generation_config=genai.GenerationConfig(max_output_tokens=200)) response = short_model.generate_content(\u0026#39;Write a 1000 word essay on the importance of olives in modern society.\u0026#39;) print(response.text) Temperature # Top-K \u0026amp; Top-P\nSimilar to temperature, the top-K and top-P parameters help manage the diversity of the model’s output.\nTop-K is a positive integer that limits the selection to the most likely tokens. Setting it to 1 results in greedy decoding, where only the most probable token is chosen.\nTop-P sets a cumulative probability limit, stopping token selection once this threshold is exceeded. A top-P of 0 mirrors greedy decoding, while a top-P of 1 allows for any token from the vocabulary.\nWhen using both parameters, the Gemini API applies top-K first, then top-P, before sampling based on the set temperature.\nmodel = genai.GenerativeModel( \u0026#39;gemini-1.5-flash-001\u0026#39;, generation_config=genai.GenerationConfig( # These are the default values for gemini-1.5-flash-001. temperature=1.0, top_k=64, top_p=0.95, )) # When running lots of queries, it\u0026#39;s a good practice to use a retry policy so your code # automatically retries when hitting Resource Exhausted (quota limit) errors. retry_policy = { \u0026#34;retry\u0026#34;: retry.Retry(predicate=retry.if_transient_error, initial=10, multiplier=1.5, timeout=300) } for _ in range(5): response = high_temp_model.generate_content(\u0026#39;Pick a random colour... (respond in a single word)\u0026#39;, request_options=retry_policy) if response.parts: print(response.text, \u0026#39;-\u0026#39; * 25) story_prompt = \u0026#34;You are a creative writer. Write a short story about a cat who goes on an adventure.\u0026#34; response = model.generate_content(story_prompt, request_options=retry_policy) print(response.text) ","date":"November 2024","externalUrl":null,"permalink":"/llm-api/gemini/gemini_api/","section":"LLM API","summary":"","title":"Gemini API 1","type":"llm-api"},{"content":"","date":"November 2024","externalUrl":null,"permalink":"/llm-api/","section":"LLM API","summary":"","title":"LLM API","type":"llm-api"},{"content":"","date":"November 2024","externalUrl":null,"permalink":"/posts/","section":"Blog","summary":"","title":"Blog","type":"posts"},{"content":" Exploring Q-learning, SARSA, Monte Carlo, and Double Q-learning: Practical Insights Beyond the Books # In the journey to mastering reinforcement learning (RL), foundational algorithms like Q-learning, SARSA, Monte Carlo Control, and Double Q-learning are essential building blocks. While many textbooks introduce these algorithms and their mechanics, several practical aspects and nuanced questions often remain unexplored. This blog post dives into these methods with hands-on insights, explaining their differences, tackling common questions, and highlighting practical challenges such as maximization bias, policy consistency, and convergence speed. Let’s delve into these four algorithms, discuss why they behave differently, and compare their performance in a \u0026ldquo;slippery walk\u0026rdquo; environment.\n# 1. SARSA and Q-learning: On-Policy vs. Off-Policy Learning # Question: Why do we pre-select an action in SARSA before entering the loop, while in Q-learning, we don’t take an initial action? # The difference between SARSA and Q-learning stems from their distinct learning paradigms: on-policy and off-policy.\nSARSA (State-Action-Reward-State-Action) is an on-policy algorithm, meaning it learns the Q-values based on the actions taken by the current policy. This requires selecting an action for each state as we proceed, ensuring that the Q-values are updated based on the policy’s actual behavior. Therefore, SARSA selects the first action before entering the loop to follow the policy consistently throughout the episode:\n$$ Q(s, a) \\leftarrow Q(s, a) + \\alpha \\left[ R + \\gamma Q(s\u0026rsquo;, a\u0026rsquo;) - Q(s, a) \\right] $$\nHere, $a\u0026rsquo;$ (the action in the next state $s\u0026rsquo;$) is selected based on the current policy, making the algorithm on-policy.\nQ-learning, on the other hand, is an off-policy method, as it updates Q-values using the maximum possible action-value in the next state, independent of the policy used to generate the experience. This allows Q-learning to approximate the optimal Q-values while following an exploratory policy. Hence, we do not pre-select an action for Q-learning, as it only needs the best Q-value in the next state:\n$$ Q(s, a) \\leftarrow Q(s, a) + \\alpha \\left[ R + \\gamma \\max_{a\u0026rsquo;} Q(s\u0026rsquo;, a\u0026rsquo;) - Q(s, a) \\right] $$\nIn summary, SARSA selects the first action to follow the current policy, while Q-learning skips this step, aiming to learn the optimal policy irrespective of the exploratory behavior.\n2. Q-learning’s Overestimation and Double Q-learning’s Solution # Question: Why does Q-learning overestimate the value function, and how does Double Q-learning address this issue? # One of the limitations of Q-learning is that it often overestimates the value function. This occurs because Q-learning’s update rule takes the maximum Q-value in the next state across all actions. By always choosing the maximum, Q-learning unintentionally amplifies any positive biases, a phenomenon known as maximization bias.\n\u0026ldquo;Q-learning often overestimates the value function. On every step, we take the maximum over the estimates of the action-value function of the next state. But what we need is the actual value of the maximum action-value function of the next state.\u0026rdquo;\nTo address this, Double Q-learning uses two separate Q-tables, Q1 and Q2, which alternate between selecting actions and evaluating them. Double Q-learning’s update rule is:\n$$ Q_1(s, a) \\leftarrow Q_1(s, a) + \\alpha \\left[ R + \\gamma Q_2(s\u0026rsquo;, \\arg\\max_{a\u0026rsquo;} Q_1(s\u0026rsquo;, a\u0026rsquo;)) - Q_1(s, a) \\right] $$\nor alternatively,\n$$ Q_2(s, a) \\leftarrow Q_2(s, a) + \\alpha \\left[ R + \\gamma Q_1(s\u0026rsquo;, \\arg\\max_{a\u0026rsquo;} Q_2(s\u0026rsquo;, a\u0026rsquo;)) - Q_2(s, a) \\right] $$\nDouble Q-learning reduces maximization bias by:\nAction Selection: Using np.argmax on one Q-table (e.g., Q1) to select the action for the next state. Action Evaluation: Evaluating the chosen action’s value using the other Q-table (e.g., Q2). This separation helps prevent the accumulation of positive bias. \u0026ldquo;Doing this isn’t only an inaccurate way of estimating the maximum value but also a more significant problem, given that these bootstrapping estimates, which are used to form TD targets, are often biased. The use of a maximum of biased estimates as the estimate of the maximum value is a problem known as maximization bias.\u0026rdquo;\nThis approach provides a more accurate Q-function estimate by reducing overestimation bias, making Double Q-learning more reliable in stochastic environments.\n3. Monte Carlo vs. Temporal Difference Methods # Question: How does Monte Carlo control differ from SARSA and Q-learning in terms of learning updates? # Unlike SARSA and Q-learning, which update Q-values after every step (using bootstrapping), Monte Carlo Control waits until the end of each episode to calculate the total return for each state-action pair and uses this to update Q-values:\n$$ Q(s, a) \\leftarrow Q(s, a) + \\alpha \\left[ G - Q(s, a) \\right] $$\nwhere $G$ is the cumulative reward from the end of the episode. Monte Carlo methods can be slower because they require full episodes to complete:\n\u0026ldquo;One of the disadvantages of Monte Carlo methods is that they’re offline methods in an episode-to-episode sense.\u0026rdquo;\nThus, Monte Carlo often has a slower convergence rate compared to SARSA and Q-learning, which use step-wise updates for faster feedback.\n4. Comparing Performance and Convergence: Experiment Results # After implementing and running all four algorithms (Monte Carlo, SARSA, Q-learning, and Double Q-learning) in a \u0026ldquo;slippery walk\u0026rdquo; environment, we observed their convergence behavior. Here’s a summary of our findings:\nQ-learning demonstrated the fastest convergence due to its off-policy nature, directly approximating the optimal policy. SARSA converged more slowly than Q-learning, as it is on-policy and must follow the exploratory policy, leading to more conservative updates. Monte Carlo Control required entire episodes for updates, which slowed down convergence compared to both SARSA and Q-learning. Double Q-learning showed the slowest convergence due to alternating updates between two Q-tables, but it mitigated maximization bias effectively. This comparison highlights that each algorithm has distinct strengths and limitations: Q-learning excels in speed, SARSA in policy adherence, Monte Carlo in complete return estimates, and Double Q-learning in bias reduction. more information on GitHub.\nPractical examples # Conclusion # In summary:\nSARSA and Q-learning balance policy fidelity and optimality through on-policy and off-policy approaches, respectively. Double Q-learning addresses maximization bias in Q-learning by separating action selection and evaluation, though it converges more slowly. Monte Carlo Control provides full return estimates per episode, beneficial when step-wise updates are impractical. Understanding these practical aspects helps navigate the complexities of real-world RL tasks, where algorithmic trade-offs like speed vs. accuracy, on-policy vs. off-policy, and bias vs. variance are critical. Each algorithm has its strengths, and the choice depends on the task\u0026rsquo;s specific needs.\nThis exploration sheds light on the nuances of RL algorithms beyond textbook explanations, providing practical insights for choosing the right method in various applications.\n","date":"November 2024","externalUrl":null,"permalink":"/posts/rl/reinforcement-learning-2/","section":"Blog","summary":"","title":"Reinforcement learning 2","type":"posts"},{"content":" Poison Theme Manual for Hugo # This guide is intended for users who are new to the Poison theme but have access to the Poison repository, Hugo, and GitHub Actions for deployment. This manual addresses key setup steps, file structure, deployment, and troubleshooting techniques.\nTable of Contents # Installing and Running Poison Locally Deploying to GitHub Pages with GitHub Actions Image Handling and Path Management File System Structure Troubleshooting Common Issues Creating New Posts Hugo.toml File Specifications 1. Installing and Running Poison Locally # Clone the repository with the Poison theme submodule. Run the following command in the terminal to fetch submodules: git submodule update --init --recursive Start the Hugo server for local development: hugo server 2. Deploying to GitHub Pages with GitHub Actions # Here\u0026rsquo;s the basic GitHub Actions setup in .github/workflows/deploy.yml to deploy on GitHub Pages:\n# Sample workflow for building and deploying a Hugo site to GitHub Pages name: Deploy Hugo site to Pages on: # Runs on pushes targeting the default branch push: branches: - main # Allows you to run this workflow manually from the Actions tab workflow_dispatch: # Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages permissions: contents: read pages: write id-token: write # Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued. # However, do NOT cancel in-progress runs as we want to allow these production deployments to complete. concurrency: group: \u0026#34;pages\u0026#34; cancel-in-progress: false # Default to bash defaults: run: shell: bash jobs: # Build job build: runs-on: ubuntu-latest env: HUGO_VERSION: 0.134.2 steps: - name: Install Hugo CLI run: | wget -O ${{ runner.temp }}/hugo.deb https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-amd64.deb \\ \u0026amp;\u0026amp; sudo dpkg -i ${{ runner.temp }}/hugo.deb - name: Install Dart Sass run: sudo snap install dart-sass - name: Checkout uses: actions/checkout@v4 with: submodules: recursive fetch-depth: 0 - name: Setup Pages id: pages uses: actions/configure-pages@v5 - name: Install Node.js dependencies run: \u0026#34;[[ -f package-lock.json || -f npm-shrinkwrap.json ]] \u0026amp;\u0026amp; npm ci || true\u0026#34; - name: Build with Hugo env: HUGO_CACHEDIR: ${{ runner.temp }}/hugo_cache HUGO_ENVIRONMENT: production TZ: America/Los_Angeles run: | hugo \\ --gc \\ --minify \\ --baseURL \u0026#34;${{ steps.pages.outputs.base_url }}/\u0026#34; - name: Upload artifact uses: actions/upload-pages-artifact@v3 with: path: ./public # Deployment job deploy: environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} runs-on: ubuntu-latest needs: build steps: - name: Deploy to GitHub Pages id: deployment uses: actions/deploy-pages@v4 3. Image Handling and Path Management # Store images in static/images or assets/images. Reference images in Markdown using relURL for consistent pathing: ![Description](\u0026amp;#123;\u0026amp;#123;\u0026amp;lt; relURL \u0026#34;images/example.jpg\u0026#34; \u0026amp;gt;\u0026amp;#125;\u0026amp;#125;) 4. File System Structure # content: Stores pages and posts. Organized into folders like posts, projects, etc. static: Contains assets like images, available site-wide. themes/custom-theme: Custom theme files. 5. Troubleshooting Common Issues # Issue with Relative Paths # Avoid leading slashes / in paths, e.g., use images/image.jpg instead of /images/image.jpg. Sidebar Image Not Showing # Check the path in hugo.toml, using brand_image = \u0026quot;images/brand_image.jpg\u0026quot;. 6. Creating New Posts # Use Hugo to create a post with front matter automatically:\nhugo new posts/new-post.md The front matter template will look like:\n--- title: \u0026#34;New Post Title\u0026#34; date: {datetime.now().strftime(\u0026#39;%Y-%m-%dT%H:%M:%S%z\u0026#39;)} draft: false --- 7. Hugo.toml File Specifications # baseURL = \u0026#34;https://username.github.io/repo/\u0026#34; languageCode = \u0026#34;en-us\u0026#34; theme = \u0026#34;poison\u0026#34; [params] brand_image = \u0026#34;images/brand_image.jpg\u0026#34; description = \u0026#34;Site description here\u0026#34; [params.menu] {Name = \u0026#34;About\u0026#34;, URL = \u0026#34;about/\u0026#34;, HasChildren = false}, {Name = \u0026#34;Posts\u0026#34;, URL = \u0026#34;posts/\u0026#34;, Pre = \u0026#34;Recent\u0026#34;, HasChildren = true, Limit = 5}, ","date":"November 2024","externalUrl":null,"permalink":"/posts/poison-setup/poison-setup/","section":"Blog","summary":"","title":"Poison setup","type":"posts"},{"content":" Monte Carlo (MC) Methods in Reinforcement Learning # Introduction to Monte Carlo (MC) Methods # Monte Carlo (MC) methods are a foundational model-free approach in reinforcement learning (RL) for estimating the value function of a given policy. The model-free nature of MC is important because it means that MC methods do not require prior knowledge of the environment\u0026rsquo;s transition dynamics or reward structure, unlike Dynamic Programming (DP) methods. Instead, MC relies on empirical data, directly sampling episodes from the environment to estimate state or state-action values.\nIn this post, we will answer common questions about MC methods, including how they work, how they differ from Temporal Difference (TD) and Dynamic Programming (DP) methods, and why they are model-free. We’ll focus on the characteristics of MC methods, including bias, variance, and bootstrapping, which often cause confusion.\nKey Concepts in Monte Carlo Methods # Episode-Based Learning:\nMonte Carlo methods learn by running complete episodes, which are sequences of states, actions, and rewards that naturally end (e.g., reaching a terminal state in a game or a defined cutoff point). By observing the cumulative return over an entire episode, MC methods avoid bootstrapping and rely on actual returns rather than estimated future rewards, which differentiates MC from TD methods. Calculating the Return:\nFor each episode, the return $( G_t )$ at a particular time step $( t )$ is calculated as: $$ G_t = R_{t+1} + \\gamma R_{t+2} + \\gamma^2 R_{t+3} + \\dots + \\gamma^{T-t-1} R_T $$ This return reflects the actual cumulative reward observed from the state at time ( t ) to the end of the episode. It is unbiased because it represents the real reward sequence under the current policy, without relying on any model. Handling States Visited Multiple Times:\nFirst-Visit MC: Only the first occurrence of each state in an episode contributes to the value function update. This approach reduces computation but still converges to the correct estimate over time. Every-Visit MC: Updates the value function for each occurrence of a state in an episode. This method averages returns over multiple occurrences, providing a more data-rich update but with potentially higher variance. Monte Carlo Update Rule: The Incremental Averaging Approach # After each episode, Monte Carlo updates the value function based on the observed returns:\n$$ V(s) \\leftarrow V(s) + \\alpha (G(s) - V(s)) $$\nwhere:\n$( V(s) )$ is the current estimate of the value function for state ( s ). $( G(s) )$ is the actual return from an episode where ( s ) was visited. $( \\alpha )$ is the learning rate, determining how much to adjust the value based on this new observation. This update formula is an incremental average of returns observed for each state. It provides a stable approximation of the expected value as episodes accumulate, allowing MC to converge to the true expected return.\nCharacteristics of Monte Carlo Methods # Model-Free:\nMonte Carlo methods are model-free, meaning they do not need the environment’s transition probabilities or reward function to calculate value estimates. This is a crucial distinction from DP methods, which are model-based. Instead, MC methods learn directly from sampling episodes and observing actual rewards, making them flexible and widely applicable. Unbiased but High Variance:\nMC estimates are unbiased because they rely on real returns (i.e., empirical observations) without estimating future values based on other states. However, MC estimates can have high variance since the returns for a given state can vary significantly from episode to episode. This variance means that Monte Carlo methods often require many episodes to stabilize the value function estimates, which can slow down learning. No Bootstrapping:\nUnlike TD methods, which bootstrap (i.e., use the value of the next state as part of the update), MC methods calculate values from complete episodes without relying on estimated future values. This can be seen as an advantage in terms of reducing potential bias but comes at the cost of requiring many episodes to average out the high variance. Comparison with Temporal Difference (TD) and Dynamic Programming (DP) Methods # Temporal Difference (TD) Methods:\nTD methods, like SARSA and Q-learning, are also model-free but bootstrap their updates. They use one-step lookahead based on the estimated value of the next state: $$ V(s) \\leftarrow V(s) + \\alpha \\left( R_{t+1} + \\gamma V(s\u0026rsquo;) - V(s) \\right) $$ By estimating values based on immediate rewards and the next state, TD methods are faster to converge but introduce a small bias due to bootstrapping. Dynamic Programming (DP) Methods:\nDP methods, such as value iteration, require a model of the environment (transition and reward functions) to calculate exact expected returns, using one-step expectations to compute the next values. DP is therefore not directly applicable to real-world problems without a known model. While DP is highly efficient when a model is available, it is limited to environments where the model’s dynamics are known in advance. Practical Applications of Monte Carlo Methods # Monte Carlo methods are well-suited for episodic tasks with a clear end state, like games or specific decision tasks in finance. However, the high variance and dependency on complete episodes can make MC less practical in continuous or real-time environments, where TD or even model-based approaches may be more effective.\nIn summary, Monte Carlo methods offer a model-free, unbiased approach to policy evaluation and improvement, learning directly from real episode returns. While they may require extensive data, their simplicity and flexibility make them valuable in a variety of reinforcement learning settings.\n","date":"October 2024","externalUrl":null,"permalink":"/posts/monte-carlo/montecarlo/","section":"Blog","summary":"","title":"Monte Carlo (MC) Methods 1","type":"posts"},{"content":" Soft Actor-Critic (SAC) Implementation # Key Points # SAC is a reinforcement learning algorithm for continuous action spaces, balancing exploration and exploitation via entropy. It uses an Actor for policy, two Q-Networks for value estimation, a Value Network for state values, and a Replay Buffer for experience. Mathematics includes Gaussian sampling, Q-value updates, and entropy regularization, crucial for training stability. Research suggests SAC performs well on benchmarks like Pendulum-v1, but results vary by environment. Introduction to SAC # Soft Actor-Critic (SAC) is an advanced reinforcement learning (RL) method designed for continuous control tasks, such as robotic arm movement or pendulum balancing. Imagine an agent exploring a vast landscape, needing to decide actions like how far to swing a pendulum. SAC helps by not just chasing rewards but also encouraging exploration, ensuring the agent doesn\u0026rsquo;t get stuck in one place.\nComponents and Mathematics # SAC\u0026rsquo;s strength lies in its components, each with a mathematical backbone:\nActor (Policy Network): Outputs a Gaussian distribution, sampling actions for exploration. It uses $\\mu(s)$ (mean) and $\\sigma(s)$ (standard deviation), transformed by $\\tanh$ for bounds.\nQ-Networks (Critics): Two networks estimate $Q(s, a)$, the value of taking action $a$ in state $s$. They minimize $(Q(s, a) - y)^2$, where $y = r + \\gamma (1 - d) \\cdot V(s\u0026rsquo;)$.\nValue Network: Estimates $V(s)$, the state value, updated to match $\\min(Q_1, Q_2) - \\alpha \\log \\pi(a|s)$, stabilizing training.\nReplay Buffer: Stores experiences $(s, a, r, s\u0026rsquo;, d)$, sampled randomly for off-policy learning.\nTesting and Application # SAC shines on benchmarks like Pendulum-v1, where it learns to balance a pendulum. Training involves episodes of actions, updates, and reward tracking, showing its adaptability.\nSurvey Note: Comprehensive Analysis of Soft Actor-Critic (SAC) # Introduction and Background # Soft Actor-Critic (SAC) is a state-of-the-art reinforcement learning (RL) algorithm, particularly effective for continuous action spaces, as seen in tasks like robotic control or pendulum balancing. Developed to address the limitations of earlier methods like Deep Deterministic Policy Gradients (DDPG), SAC introduces entropy regularization to balance exploration and exploitation, enhancing stability and sample efficiency. This survey note, informed by our detailed exploration, aims to provide a thorough understanding of SAC\u0026rsquo;s components, mathematics, and practical application, drawing on online references and addressing common queries.\nSAC Components and Their Roles # SAC comprises five key components, each with a specific role in the learning process:\nActor (Policy Network): The Actor defines the policy $\\pi(a|s)$, outputting a Gaussian distribution over actions to enable stochastic behavior. This is crucial for exploration, as it allows the agent to try diverse actions rather than sticking to deterministic choices. The policy is parameterized by mean $\\mu(s)$ and standard deviation $\\sigma(s)$, often represented as log standard deviation for numerical stability.\nQ-Networks (Critics): SAC employs two Q-Networks, $Q_1(s, a)$ and $Q_2(s, a)$, to estimate the action-value function. The use of two networks mitigates overestimation bias, a common issue in Q-learning, by taking the minimum of their predictions. This dual-critic approach enhances training stability, especially in continuous spaces.\nValue Network: The Value Network estimates the state-value function $V(s)$, which represents the expected return starting from state $s$ under the current policy, including an entropy term. It plays a pivotal role in providing stable targets for the Q-Networks, reducing variance in updates and ensuring consistent learning.\nReplay Buffer: Essential for off-policy learning, the Replay Buffer stores experiences $(s, a, r, s\u0026rsquo;, d)$ and allows random sampling. This breaks temporal correlations, improving data efficiency and training stability by reusing past experiences.\nSAC Agent: The agent integrates all components, managing environment interactions, experience storage, and network updates. It orchestrates the training loop, ensuring coordinated updates to the Actor, Critics, and Value Network, with soft updates to the target Value Network for additional stability.\nMathematical Foundations # The mathematics of SAC is central to its operation, and we\u0026rsquo;ll explore each component\u0026rsquo;s equations:\nActor\u0026rsquo;s Sampling and Log Probability: # The Actor outputs $\\mu(s)$ and $\\log \\sigma(s)$, computing $\\sigma(s) = e^{\\log \\sigma(s)}$. Actions are sampled from $\\mathcal{N}(\\mu(s), \\sigma(s)^2)$, then transformed: $a = \\tanh(u)$, where $u \\sim \\mathcal{N}(\\mu, \\sigma)$. The log probability is adjusted for the $\\tanh$ transformation: $\\log \\pi(a|s) = \\log \\pi(u|s) - \\sum_{i=1}^{d} \\log(1 - a_i^2 + \\epsilon)$ where $\\log \\pi(u|s) = -\\frac{1}{2} \\sum_{i=1}^{d} \\left[ \\left(\\frac{u_i - \\mu_i}{\\sigma_i}\\right)^2 + 2 \\log \\sigma_i + \\log 2\\pi \\right]$, and $\\epsilon = 10^{-6}$ prevents numerical issues. Q-Networks\u0026rsquo; Update: # Trained to minimize the Bellman error: $L_Q(\\theta) = \\mathbb{E} \\left[ \\left( Q_\\theta(s, a) - y \\right)^2 \\right]$ where the target $y = r + \\gamma (1 - d) \\cdot V(s\u0026rsquo;)$, with $\\gamma$ as the discount factor and $d$ as the done flag. Value Network\u0026rsquo;s Update: # Updated to match: $V(s) = \\mathbb{E}_{a \\sim \\pi} \\left[ \\min(Q_1(s, a), Q_2(s, a)) - \\alpha \\log \\pi(a|s) \\right]$ This incorporates entropy regularization, controlled by $\\alpha$, ensuring exploration. Actor\u0026rsquo;s Update: # The Actor maximizes: $J(\\pi) = \\mathbb{E}_{s \\sim \\mathcal{D}, a \\sim \\pi} \\left[ Q(s, a) - \\alpha \\log \\pi(a|s) \\right]$ This encourages actions with high Q-values while maintaining entropy for exploration. Implementation Highlights # To illustrate, here are key code snippets for critical components:\nActor\u0026rsquo;s Sampling: # def sample(self, state, deterministic=False): mu, log_std = self.forward(state) std = torch.exp(log_std) if deterministic: action = torch.tanh(mu) else: dist = torch.distributions.Normal(mu, std) u = dist.rsample() action = torch.tanh(u) log_prob = dist.log_prob(u) - torch.log(1 - action.pow(2) + 1e-6) return action, log_prob This shows action sampling with the reparameterization trick and log probability adjustment.\nQ-Network Forward Pass: # def forward(self, state, action): x = torch.cat([state, action], dim=-1) return self.q_net(x) Demonstrates how Q-Networks process concatenated state-action pairs.\nReplay Buffer Sampling: # def sample(self, batch_size=None): batch_size = batch_size or self.batch_size indices = np.random.choice(self.size, size=batch_size, replace=False) return ( torch.tensor(self.states[indices], dtype=torch.float32, device=self.device), torch.tensor(self.actions[indices], dtype=torch.float32, device=self.device), torch.tensor(self.rewards[indices], dtype=torch.float32, device=self.device), torch.tensor(self.next_states[indices], dtype=torch.float32, device=self.device), torch.tensor(self.dones[indices], dtype=torch.float32, device=self.device) ) Highlights random sampling for off-policy learning.\nAnticipated Questions and Answers # During teaching, students often ask:\nWhy two Q-Networks?\nUsing two reduces overestimation bias by taking the minimum, enhancing stability. It\u0026rsquo;s like having two advisors to cross-check decisions.\nWhat is entropy regularization and why is it important?\nIt encourages exploration by rewarding random actions, preventing the agent from getting stuck. It\u0026rsquo;s like ensuring the agent doesn\u0026rsquo;t always take the same path, fostering discovery.\nHow does the replay buffer help in training?\nIt stores past experiences, allowing off-policy learning. Random sampling breaks temporal correlations, stabilizing updates, like learning from a diverse history.\nWhat is the reparameterization trick and why is it used?\nIt rewrites sampling as $u = \\mu + \\sigma \\cdot \\epsilon$, where $\\epsilon \\sim \\mathcal{N}(0, 1)$, enabling gradient flow through $\\mu$ and $\\sigma$, essential for policy optimization.\nTesting and Application # To validate SAC, we tested it on Pendulum-v1, a benchmark from Gym, where the agent learns to balance a pendulum. The setup involved:\nState dimension: 3 (angle, angular velocity, etc.). Action dimension: 1 (torque). Training for 1000 episodes, monitoring cumulative rewards. This practical application demonstrated SAC\u0026rsquo;s ability to adapt, with rewards improving over time, showcasing its effectiveness in continuous control.\n","date":"April 2023","externalUrl":null,"permalink":"/posts/sac/sac/","section":"Blog","summary":"","title":"Soft Actor-Critic (SAC)","type":"posts"},{"content":"","date":"March 2023","externalUrl":null,"permalink":"/tags/computer-vision/","section":"Tags","summary":"","title":"Computer Vision","type":"tags"},{"content":"","date":"March 2023","externalUrl":null,"permalink":"/tags/datasets/","section":"Tags","summary":"","title":"Datasets","type":"tags"},{"content":"","date":"March 2023","externalUrl":null,"permalink":"/tags/depth-estimation/","section":"Tags","summary":"","title":"Depth Estimation","type":"tags"},{"content":"Research and engineering work, each backed by its code. Energy and reinforcement learning projects are being added.\n","date":"March 2023","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"","title":"Projects","type":"projects"},{"content":"","date":"March 2023","externalUrl":null,"permalink":"/tags/pytorch/","section":"Tags","summary":"","title":"PyTorch","type":"tags"},{"content":"","date":"June 2022","externalUrl":null,"permalink":"/tags/object-detection/","section":"Tags","summary":"","title":"Object Detection","type":"tags"},{"content":"","date":"June 2022","externalUrl":null,"permalink":"/tags/robotics/","section":"Tags","summary":"","title":"Robotics","type":"tags"},{"content":"","date":"June 2022","externalUrl":null,"permalink":"/tags/tensorflow/","section":"Tags","summary":"","title":"TensorFlow","type":"tags"},{"content":"","date":"June 2022","externalUrl":null,"permalink":"/tags/tracking/","section":"Tags","summary":"","title":"Tracking","type":"tags"},{"content":"I\u0026rsquo;m a research engineer at Honda Research Institute Europe in Offenbach and a PhD candidate in electronics engineering at Sapienza University of Rome, defending in October 2026.\nMy work is deep reinforcement learning for sequential decision making under uncertainty: continuous and discrete action spaces, multi-objective and constrained formulations, and multi-agent settings. In the PhD I applied it to battery dispatch and real-time energy management for renewable energy communities, with a particular interest in policies that account for battery degradation rather than optimising cost alone. At Honda I work on multi-agent coordination under shared resource constraints, currently fair electric-vehicle charging under a shared power budget, where an explicit fairness constraint replaces hand-tuned reward weights. In between, as a guest researcher at TU Delft, I built a reinforcement learning environment for the long-term energy transition of the Netherlands at national scale.\nBefore energy systems I worked in computer vision and robotics at EPFL\u0026rsquo;s VITA Lab: deep stereo networks for depth estimation on unrectified 360-degree images, a new lidar-and-camera dataset, and a real-time person detection and tracking pipeline running on a Loomo mobile robot.\nI write mostly in Python and PyTorch, with C/C++, MATLAB/Simulink and ROS where the hardware needs it.\nSee the CV, publications and projects. I\u0026rsquo;m available from November 2026, and LinkedIn is the best place to reach me.\n","externalUrl":null,"permalink":"/about/","section":"Danial Zendehdel","summary":"","title":"About","type":"page"},{"content":"Research engineer at Honda Research Institute Europe and PhD candidate in electronics engineering at Sapienza University of Rome, defending in October 2026. Specialised in deep reinforcement learning for sequential decision making under uncertainty, in continuous and discrete action spaces and with multi-objective, constrained and multi-agent formulations. Applied to battery dispatch and energy management systems in the PhD, and to multi-agent coordination under shared resource constraints at Honda. Earlier work in computer vision and robotics at EPFL, including deep stereo networks for 360-degree depth estimation and a real-time person detection and tracking pipeline deployed on a mobile robot.\nAvailable from November 2026. LinkedIn · GitHub · Publications · Projects\nExperience # Honda Research Institute Europe GmbHJun 2026 – present Research EngineerOffenbach, Germany Trained multi-agent deep reinforcement learning policies for electric vehicle charging under a shared power budget; reached a wider Pareto front on efficiency and fairness than the group\u0026rsquo;s evolutionary (genetic programming) schedulers. Replaced hand-tuned reward weights with an explicit fairness constraint in actor-critic policy gradient training, giving an interpretable and verifiable fairness guarantee between agents for trustworthy deployment. Built the simulation environment, cost model and evaluation pipeline for benchmarking against optimisation-based and evolutionary baselines under the group\u0026rsquo;s published protocol. TU DelftSep 2025 – Apr 2026 Guest ResearcherDelft, Netherlands Wrote a reinforcement learning environment and trained agents in it to simulate the long-term energy transition of the Netherlands at national scale. Set up multi-year investment and retrofitting choices under uncertainty as a sequential decision problem, covering renewable integration and system planning. Sapienza Università di RomaNov 2023 – Oct 2026 PhD Researcher, Electronics EngineeringRome, Italy Trained deep reinforcement learning agents for battery dispatch in a real-time energy management system (EMS) of a renewable energy community, in both continuous and discrete action spaces; achieved lower operating cost than forecasting-based and optimisation baselines. Reduced battery degradation at comparable operating cost by writing multi-objective rewards that trade the two against each other; published as a degradation-aware RL framework (IJCCI 2025) and an enhanced SAC algorithm (IJCNN 2025, IEEE). Built the simulation environment around an extended battery equivalent-circuit model, giving the battery management system (BMS) side of the EMS a physical basis (IJCNN 2024, IEEE). Compared seasonal and non-seasonal forecasting methods for energy demand time series in smart grids and selected the forecaster used by the EMS (IJCCI 2023). École Polytechnique Fédérale de Lausanne (EPFL)Feb 2022 – Aug 2022 Visiting Researcher, VITA LabLausanne, Switzerland Modified the architecture of existing deep stereo networks so they work on unrectified 360-degree camera images, after first reproducing the published benchmark results on the original datasets. Built a data collection rig from a lidar and two 360-degree cameras, recorded a new dataset, debugged the lidar extraction, and released the cleaned dataset in a standard format for the group. Ran a person detection and tracking pipeline on a Loomo mobile robot, with all inference on the onboard computer, so the robot followed a chosen person in real time. Education # Sapienza Università di RomaNov 2023 – Oct 2026 PhD in Electronics Engineering, defence scheduled October 2026Rome, Italy Thesis: Smart Control of Renewable Energy Sapienza Università di RomaSep 2019 – Dec 2022 MSc in Electronics Engineering, cum laudeRome, Italy Thesis: Depth Estimation with Stereo 360-Degree Cameras (EPFL VITA Lab) · project page Babol Noshirvani University of TechnologySep 2013 – Sep 2018 BSc in Electrical EngineeringBabol, Iran Skills # Reinforcement learning: deep RL in continuous and discrete action spaces, multi-objective and constrained RL, multi-agent RL, actor-critic and policy gradient methods, reward design, simulation environment design\nComputer vision and perception: stereo depth estimation, 360-degree and omnidirectional imaging, object detection and tracking, lidar and camera data collection and calibration, dataset construction and cleaning, real-time inference on embedded hardware\nMachine learning: deep learning, time series forecasting, PyTorch, TensorFlow\nOptimisation and control: multi-objective and constrained optimisation, linear and mixed-integer programming, modelling and simulation of dynamic systems\nEnergy systems: energy management systems, battery management systems, battery dispatch, degradation modelling, renewable energy communities, smart grids\nProgramming and tools: Python, C/C++, MATLAB/Simulink, Git, Linux, bash, ROS\nRobotics and embedded: NVIDIA Jetson, Raspberry Pi, Arduino, FPGA digital design (VHDL)\nLanguages: English (fluent), Persian (native), Italian (A2), French (A2)\nPublications # Game-Theoretic Fair Allocation in MPC-Based Management of Renewable Energy Communities D. Zendehdel Accepted for publication, 2026 Accepted Forward–Forward Learning for Imbalanced Tabular Predictive Maintenance on a Real-World Smart-Grid Fault Dataset E. De Santis, D. Zendehdel, G. Ferro, A. Rizzi Manuscript, 2026 Manuscript Intelligent Energy Management for Renewable Energy Communities and Microgrids: Models, Algorithms, and Practical Constraints D. Zendehdel, G. Ferro, E. De Santis, A. Rizzi Stochastic Planning and Modeling for Energy Systems, Elsevier, pp. 491–532, 2026 Book chapterDOI Degradation-Aware Energy Management in Residential Microgrids: A Reinforcement Learning Framework D. Zendehdel, G. Ferro, E. De Santis, A. Rizzi 17th International Joint Conference on Computational Intelligence (IJCCI 2025), Springer CCIS, pp. 538–557, 2025 ConferenceDOI Multi-Objective Battery Dispatching using an Enhanced SAC Algorithm D. Zendehdel, E. De Santis, A. Capillo, P. Odonkor, A. Rizzi International Joint Conference on Neural Networks (IJCNN 2025), IEEE, pp. 1–8, 2025 ConferenceDOI Seasonality Effect Exploration for Energy Demand Forecasting in Smart Grids S. Taghdisi Rastkar, D. Zendehdel, A. Capillo, E. De Santis, A. Rizzi IJCCI 2023 extended version, Computational Intelligence, Studies in Computational Intelligence, Springer, pp. 211–223, 2025 Book chapterDOI An Extended Battery Equivalent Circuit Model for an Energy Community Real Time EMS D. Zendehdel, A. Capillo, E. De Santis, A. Rizzi International Joint Conference on Neural Networks (IJCNN 2024), IEEE, pp. 1–9, 2024 ConferenceDOI A Comparison Between Seasonal and Non-Seasonal Forecasting Techniques for Energy Demand Time Series in Smart Grids S. Rastkar, D. Zendehdel, E. De Santis, A. Rizzi 15th International Joint Conference on Computational Intelligence (IJCCI 2023), SciTePress, pp. 459–467, 2023 ConferenceDOI Full CV with contact details available on request via LinkedIn.\n","externalUrl":null,"permalink":"/cv/","section":"Danial Zendehdel","summary":"","title":"CV","type":"page"},{"content":"Peer-reviewed conference papers and book chapters from my PhD at Sapienza University of Rome, supervised by Prof. Antonello Rizzi. My MSc thesis is listed at the end. DOI links go to the publisher\u0026rsquo;s page.\n2026 Game-Theoretic Fair Allocation in MPC-Based Management of Renewable Energy Communities D. Zendehdel Accepted for publication Accepted Forward–Forward Learning for Imbalanced Tabular Predictive Maintenance on a Real-World Smart-Grid Fault Dataset E. De Santis, D. Zendehdel, G. Ferro, A. Rizzi Manuscript Manuscript Intelligent Energy Management for Renewable Energy Communities and Microgrids: Models, Algorithms, and Practical Constraints D. Zendehdel, G. Ferro, E. De Santis, A. Rizzi Stochastic Planning and Modeling for Energy Systems, Elsevier, pp. 491–532 Book chapterDOI 2025 Degradation-Aware Energy Management in Residential Microgrids: A Reinforcement Learning Framework D. Zendehdel, G. Ferro, E. De Santis, A. Rizzi 17th International Joint Conference on Computational Intelligence (IJCCI 2025), Springer CCIS, pp. 538–557 ConferenceDOI Multi-Objective Battery Dispatching using an Enhanced SAC Algorithm D. Zendehdel, E. De Santis, A. Capillo, P. Odonkor, A. Rizzi International Joint Conference on Neural Networks (IJCNN 2025), IEEE, pp. 1–8 ConferenceDOI Seasonality Effect Exploration for Energy Demand Forecasting in Smart Grids S. Taghdisi Rastkar, D. Zendehdel, A. Capillo, E. De Santis, A. Rizzi IJCCI 2023 extended version, Computational Intelligence, Studies in Computational Intelligence, Springer, pp. 211–223 Book chapterDOI 2024 An Extended Battery Equivalent Circuit Model for an Energy Community Real Time EMS D. Zendehdel, A. Capillo, E. De Santis, A. Rizzi International Joint Conference on Neural Networks (IJCNN 2024), IEEE, pp. 1–9 ConferenceDOI 2023 A Comparison Between Seasonal and Non-Seasonal Forecasting Techniques for Energy Demand Time Series in Smart Grids S. Rastkar, D. Zendehdel, E. De Santis, A. Rizzi 15th International Joint Conference on Computational Intelligence (IJCCI 2023), SciTePress, pp. 459–467 ConferenceDOI 2022 A Deep Learning Architecture for Enhanced Depth Estimation on an Unrectified Stereo 360° Dataset D. Zendehdel MSc thesis, Sapienza University of Rome (with EPFL VITA Lab) ThesisCode ","externalUrl":null,"permalink":"/publications/","section":"Danial Zendehdel","summary":"","title":"Publications","type":"page"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]