dotnet.work

Use Claude AI in Home Assistant Studio

dotnet@oles.cloud (Ole Albers) — Wed, 04 Feb 2026 18:17:00 +0100

Sometimes you have to configure stuff in Home Assistant the “painful” way: Working with Yaml-Files instead of using the UI. Therefore Studio Code Server is the first choice for most of us. But Yaml isn’t fun. Yaml in hass.io especially isn’t fun.

But there is ai for the rescue. AI isn’t always the solution, but working with ugly configuration formats is exactly where it is good at. So let’s add claude to the Studio Code Server

Studio Code Server

I assume you already have installed the Studio Code Server. If you haven’t: It is a simple Add-On. Just follow the official instructions: https://github.com/hassio-addons/addon-vscode/blob/main/vscode/DOCS.md

Studio Code Server will then be installed as a docker container. (Don’t worry. No need to know anything about docker. This works in the background)

Preparing

Claude needs NodeJs which isn’t in the studio code server docker. But don’t worry, this is easy to solve. You could connect to the docker and install the required packages using app-get. But while this works this has two disadvantages. First it is a bit more complicated if you are not at home with Linux Systems and most important: Any update of the Addon would kill NodeJs and you would have to do this all over again.

Thankfully the Add-On devs thought of an easy way to do that for you. Just go to Settings - Add-Ons - Studio Code Server. There go to Configuration and in the “packages” enter “npm” and then “nodejs”. Do not enter both at the time. Each one of them by enter. After that there should be two bubbles on top:

If you prefer editing in yaml, here is your content:

packages:
  - nodejs
  - npm
init_commands: []

Save and restart the addon. Don’t be alarmed if this takes a LONG time. There will have big packages to be downloaded and installed into that container. This will take minutes even on a fast machine and especially if using a Raspberry Pi.

Adding Claude

Now you can switch to the code server. Select Extensions on the left side, enter “Claude” into the search and install “Claude Code for VS Code”. After that you should have that typical “Claude Star” on the top right corner. You will be greeted with the authentication method. Select the one that matches your account. (Most likely Claude.ai Subscription)

If after using the Login clicking “Open” does not work because you get redirected to a local url use the url instead that is shown in the textbox:

Just copy paste that url manually and paste the returning code into Studio Code.

Using Claude

Now you can open Claude using the star and let him do the work

That is it. Now you can use claude in Home Assistant Config Files

Vibe Coding: An experiment

dotnet@oles.cloud (Ole Albers) — Sat, 18 Oct 2025 07:48:00 +0200

Before I begin to even start writing about this topic I assume it makes sense to mention that this article has not been created using AI. Not a single word. (with one exception: The german version has been translated automatically using Claude, like all entries on my blog) I also tried to write this for a wider audience than my usual articles. As this isn’t a specific problem solving for a weird use case in c# I tried to keep the tech gibberish down and especially do not expect you to have C# knowledge. Any coding experience helps, but isn’t required. On the other hand if you are a skilled dev, don’t be annoyed if I explain things trivial in your opinion; Just ignore the explanations and feel smart :)

Ruleset

I simply wanted to test how good vibe coding (Letting the AI create your whole application) is today. Does it achieve what I want? Is the code quality good? Would it help someone who has no knowledge at all? So I set up a ruleset:

Make sure that what I want to achieve hasn’t be done before (so the AI can’t just copy/paste a complete application)
DO NOT CODE: Really. I try to not change a single character in a source file even if a mistake is obvious
Try not to help: Of course if something is wrong, I will tell the AI. But let it find out how to solve this, first only helping if it got really stuck
Application should not be too ease (no “Hello World”)
Application should not be too complecated either (no “Please create a better Microsoft Teams application”)
No external tools needed; So no Unity or Photoshop requirements, etc. I should not need to leave the IDE (Rider)
No Security related stuff: No Password hashing, not storing important secrets, simply because I trust AI enough to do this lightly. This might be worth another experiment on its own

Environment and Job description

I used Rider with Claude AI integration using Sonnet 4. Whenever it asked for a permission I granted it. I started by giving a simple task and iterate over time. The task was:

Create a Razor Application that offers a SCRUM Poker functionality with a twist: It should have Joker Mechanics like in Balatro. The PO should be able to create a new game. Then an admin view and a player view should be created. Each of them by generating a random code like play?code=dbddhkp and admin?code=ukule2 . No database. Just store everything in memory

I was sure this hasn’t been done before because it absolutely makes no sense. SCRUM poker is a tool to determine the “cost” (or complexity) of programming tasks. Each developer decides how complex the task at hand is usually using values from 1 to 21 (as always, there are a lot of variants with other numbers) and at the end these values gets compared and discussed. The main reason is that the number itself isn’t communicated openly at the beginning so if a senior dev tells his number other devs might just say “yepp. About the same”. The “poker” element makes sure everyone says the number he or she really thinks of and afterwards any discrepancies gets discussed.

Balatro on the other hand is a fun Poker game. There are Jokers that change the value of the cards. E.G a “every heart gets multiplied by 4”. Obviously this would destroy any useful results that the SCRUM poker produced. So a nonsens project, a quite complex one nonetheless.

I added additional features lateron like a multi-language feature but did want to give the AI the chance to iterate

Did it work?

The short answer is: Yes. I did not write a single line of code and the application worked.

But the long answer is: Developers wont loose their job very soon. As I will show in the learnings it requires a skilled developer to lead into the right direction when problems need to be solved. In addition some decision lead to bad code. The detailed pitfalls and also positive surprises can be found in Learnings.

(And also I published the code on GitHub so you can see for yourself)

The biggest benefit: I am a backend developer. My programs (usually) work, but I am not really good in UI or UX. That benefit the AI was able to solve. The application wasn’t the most beautiful thing ever to be seen in a browser window; Still: It looked way better than anything I would have created

Sources and demonstration

You can find the sources on the FiboNuts Repository on Github. A working example can be found at Poker.Oles.Cloud

Learnings

Code quality

Code quality started good, but got worse, completely wrong assumptions leading to bad code. Forcing Claude to revert these things helped putting the code back in good shape. TBH I expected way worse

Help needed

Quite at the beginning of the project I ran into a problem that could not be solved by the AI itself: There are URLs generated for the players. But when I enter those in a different browser (or even tab) the code does not get recognized. I tried to tell Claude in about 10 Iterations that it does not work. The AI did try different approaches but never came near a solution. After I told him that there is a problem in the Dependency Injection implementation and the class holding the state should not be added Scoped but added as a Singleton it was able to continue. But without my help this would be the end of the project.

It was surprising to see that such quite simple cause could not be recognized while much more complex issues got resolved.

Also at one point the communication between the server and client did not work and it seems like Claude simply could not believe when I said that. I had to offer assistance telling him to create logs that I can then paste back so Claude can analyze the issue.

Adding random stuff

Claude also decided at one point to add https. (It was http before) when I was asking to add a multi language feature. I assume he learned implementing that feature from a site where https was used. The https-part completly broke the application because ports had been assigned multiple times in the config.

Also until the very end there always was an ugly “About” link on the upper part of the page linking to https://learn.microsoft.com/en-us/aspnet/core/?view=aspnetcore-9.0. I assume this is part of any “how to create a Blazor Application” Tutorial and so Claude thought this is required

Wrong assumptions

At one point I asked the AI to add a multi language feature using best practices. While it did that correctly it forgot to add a crucial Package from NuGet. The Build errors have been quite obvious stating ‘Are you missing a reference’ but Claude did not seem to understand that. Instead it assumed that the code generated was too complex for the compiler (it absolutely wasn’t: Just a few lines adding language-strings to an array) and simply removed its code over and over again until the program compiled again but also had no language code left. Again I had to help and tell him: “Hey. That wasn’t the issue. You forgot some NuGet Packages”. After that the issue was resolved.

At some other point Claude managed to break the application completely. Right at the start it simply would not load anymore after the progress bar was finished. Claude did an assumption no dev would do: It was sure the cause was the logging. Too many log entries causing the application to slow down. So any log in any exception handling or anywhere else got removed. Not solving the issue, obviously but creating a way worse variant than before.

As a rule of thumb: Whenever the Bots writes something like “this is a known issue” it probably found some very weird edge case that has nothing to do with a problem that should be simple. In my case multi language not working caused really weird actions: Creating dlls for each language, trying to use reflection to fix some (non-existing) issue of the localizer when at the end the text in the razor page was simply static. After I told that there should be variables or method calls instead of fixed strings Claude was able to revert all the nonsense changes, but without that help this was unsolvable.

What is considered easy?

Tasks that I see as quite complicated have been solved sometimes without any issues while others, that shouldn’t be too hard seemed to be unsolvable. For example I wanted a ‘/metrics’ endpoint to be used with Prometheus (A tool to collect metrics and display them in a Dashboard). While it looked right, it was a html page that rendered the content. Claude did never manage to follow the simple task: “please output plain text. No html tags.” Even when providing the correct Mimetype it still did not understand correctly. An additional Server application was created where the clients sent metrics, suddenly a IIS Express configuration appeared in the launchsettings and other things that made the program way, way worse but did not even move an inch into the right direction.

After that, Claude was so confused that not even rolling back worked. The AI wasn’t able to find its files anymore and even tried to find it in regions of my computer where it isn’t supposed to look:

Needless to say that these operation have been costly:

After a few retries I had to surrender. Claude was never able to create a simple output in text/plain. Even if I gave example snippets. It always rendered HTML that looked like Text but still had html tags.

Interesting tactics

After breaking the application when adding https claude did some interesting things. It did not simply analyze the code but was sure it wasn’t its fault :) Instead it assumed there must be some other application causing the issue and so started nmap to analyze what other services on my machine might be to blame. It did not expect it to “break out” of my IDE. A reminder to not just allow everything but have a close look on all actions. I would not be surprised if it had decided to simply kill that other process if it really had been in external issue. (But the implementation explicitly asks for your permission whenever it does something it hasn’t before)

Conclusion/Opinion

Useful, but needs to be monitored closely. And you must be aware that the code isn’t really “good”, just working. Just a minor issue in this application but when customer data is processed and stored bad design decisions can lead to big problems. There are already a lot of vibe coded applications, used by millions that are just really badly written and leak data all the time. Just one example of many: https://cybernews.com/security/ai-girlfriend-app-leak-exposes-400k-users/

No developer wants to be responsible for a catastrophe like this even before thinking of GDPR fines.

Also: Even more like in human developing: Use Source Control. Claude can create git commits without any problems. This helps rolling back to previous state if the AI created a mess after an error. But I would suggest to only use a local repository and push manually to be able to make sure no secrets end up in the code. Also don’t just check the current code but any commit if there was some secret at any time.

Nonetheless we also can’t simply ignore AI when creating programs. Even if the AI bubble at some day bursts the tools will remain. LLMs will still be there and be just any other tool on the Cloud or maybe even on local machines. No matter if someone makes big cash selling AI tools or not. So in my opinion a senior dev has to decide where it works and where it does not. Also because of the nature of AI : I will get better, but there will always be the same kind of errors you can’t get rid of. Hallucinations are a thing and always will be.

I also think the next generation(s) will be the ones that benefit the most but also suffer the most because of AI. Benefit because it never was so easy to start. Need something done in your OS? Ask the AI to write a python script. Having trouble about Dependy Injection? The AI will provide a document with good examples. You could find that without AI, but that is just simpler and faster. On the negative side: Your problem gets solved, but maybe not in the best way. And this isn’t just an academic argument: You have to understand why we do not store passwords but use hashes instead to understand the underlying risk.

And in addition from now on not much more content will be created. You get every information you need through AI. But when no new information is created on the internet in a few years AI will mostly scrape content it created itself leading to a perfect endless confirmation bias loop.

And on a personal note: On the one hand: AI is a great tool, on the other hand it nearly already destroyed any use of searching the web. Have you ever tried to find information on “Is it ok if my dog eats stuff”? The first two search pages mostly contain AI generated content. And these range from “No problem. stuff is even very healthy” to “Never ever! One bite and your dog is dead!”. I got around this by using DuckDuckGo and the “Block all results from this site” - feature for a while but am at a point where sites with random domains get created faster than I cen add them to the list.

Maybe I should use AI to keep these lists up to date…. 😀

Grafana - Notifications for Mastodon

dotnet@oles.cloud (Ole Albers) — Tue, 14 Oct 2025 12:28:00 +0200

Grafana is a very useful tool for monitoring your own programs, discovering silly bugs, or simply looking at beautiful graphs :)

Another feature is “Alerts”. These can be configured arbitrarily, whether high CPU load, too many errors in the log, or similar should trigger an alarm.

This post won’t be about the alerts themselves, but rather about sending already configured alerts to Mastodon. These are called Contact Points in Grafana and initially don’t offer Mastodon integration. However, this can be worked around with a few simple steps.

Mastodon

First, we need some preparation, because of course not just any random application can post to Mastodon under our name (thankfully).

Therefore, we log into the instance of our choice, select Settings and then Development. This should look something like this:

(If the list is empty, it means you haven’t created any applications yet. If it’s not empty, you don’t need this part of the tutorial at all, because you’ve already done this before 😀)

Click on New Application here and choose a name. Leave the other two text fields as they are:

For permissions, you only need “write:statuses”.

You should now see a page showing the secrets. You only need the access token (bottom value) from this.

Grafana

Now we can switch to Grafana. There’s a menu item called Contact points under Alerting. At least Email should already appear in the list. We now add a new destination with "+ Create contact point":

The name is of course freely selectable, for integration choose Webhook

For the URL, enter the following:

https://chaos.social/api/v1/statuses?access_token=<Secret>

Instead of , enter the access token from Mastodon. If you now click on Test (and leave the default values as they are), a new entry will already be created in Mastodon under your name:

Getting Down to Business

That’s nice for starters, but we obviously can’t do much with just “firing”. We want to see the actual error. Also, we probably don’t want to inform the whole world. It makes more sense to simply mention ourselves and set the visibility to “direct”.

To do this, we open Optional Webhook Settings, set HTTP Method to POST and click the “Edit Payload Template” button next to Custom Payload. A new window pops up and after we’ve selected Enter custom payload template there, we can enter the following content:

{
  "status": "Alert: {{ .CommonLabels.alertname }}\nStatus: {{ .Status }}\n{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}"
}

Now the alert is already a bit more meaningful:

Still missing the private mode. This can be set via Visibility. Additionally, it’s best to mention yourself. (I’ve modified my name slightly here so you don’t make a copy-paste mistake :) )

While we’re at it, I’ve also used the spoiler-text.

{
  "status": "@guacamoleeeee {{ .CommonAnnotations.summary }}",
  "visibility": "direct",
  "sensitive": false,
  "spoiler_text": "🚨 {{ .CommonLabels.alertname }} {{ .Status }}",
  "language": "en"
}

The result is then as follows:

A complete list of Mastodon properties can be found here: https://docs.joinmastodon.org/methods/statuses/

Have Fun

Wishing you many alerts might be a bit strange, but I hope I could help with this.

Led Progressbar for 3d Printing

dotnet@oles.cloud (Ole Albers) — Tue, 06 May 2025 14:02:00 +0200

3D printing can be fun, but can also be very annoying if something bad happens. So I wanted to have an easy indicator if something goes wrong without having to check the print every few minutes

WLED to the rescue

My 3D - Printer is in a small room. I usually do not have to enter that room until I want to remove the print. So I decided to add an RGB-LED-Strip that shows how the print is running. I decided to use WLED using a cheap ESP8266MOD. Installing the WLED-Software is easy. It can even be done using a simple browser if you are on Windows or Mac: https://install.wled.me/

Home Assistant

After that I needed to write some automations. The WLED is detected automatically on Home Assistant, for my Creality 3d Printer I used the “Moonraker” Integration available via HACS: https://moonraker-home-assistant.readthedocs.io/en/stable/index.html

Error display

The printer reports its Print state through the “sensor.k2plus_b4f8_current_print_state” (yours can have a different prefix). The following values are possible: standby, printing, paused, complete, cancelled, error But errors are never reported as an “error” but always as “paused”. There might be more critical ones that are reported as error but I never had those. So all we have to do is to react on the “paused” - state and set the WLED accordingly:

alias: 3DP Error
description: ""
triggers:
  - entity_id:
      - sensor.k2plus_b4f8_current_print_state
    trigger: state
    to: paused
conditions: []
actions:
  - target:
      entity_id: light.wled_2
    action: light.turn_on
    data:
      effect: Breathe
      rgb_color:
        - 204
        - 0
        - 0
mode: single

Again: Your Entities might have different names

Progress display

To be able to display the progress we need the progress-Entity and in addition make sure that the printer really is printing:

alias: 3DP Progress
description: >-
  Updates WLED2 fill level to match sensor.k2plus_b4f8_progress value with
  purple color
triggers:
  - entity_id: sensor.k2plus_b4f8_progress
    trigger: state
conditions:
  - condition: state
    entity_id: sensor.k2plus_b4f8_current_print_state
    state: printing
actions:
  - target:
      entity_id: light.wled_2
    data:
      rgb_color:
        - 128
        - 0
        - 128
      effect: Percent
    action: light.turn_on
  - target:
      entity_id: number.wled_intensitat_2
    data:
      value: >
        {% set progress_value = states('sensor.k2plus_b4f8_progress') | float(0)
        %} {% if progress_value > 100 %}
          100
        {% elif progress_value < 0 %}
          0
        {% else %}
          {{ progress_value | round(1) }}
        {% endif %}
    action: number.set_value

Display Countdown

Last but not least: Because the 3d printer consumes quite an amount of power even on standby I want to shutdown power automatically when he finished printing. Shutting down itself is quite simple:

alias: 3DP Strom automatisch aus
description: ""
triggers:
  - trigger: state
    entity_id:
      - sensor.k2plus_b4f8_current_print_state
    to: complete
    for:
      hours: 0
      minutes: 15
      seconds: 0
conditions:
  - condition: numeric_state
    entity_id: sensor.k2plus_b4f8_hotend_fan
    below: 10
  - condition: numeric_state
    entity_id: sensor.k2plus_b4f8_extruder_temperature
    below: 50
actions:
  - type: turn_off
    device_id: 710995d98fe3640e134ae275e94dc491
    entity_id: 7cd829033182474ac8f91a7c11539dfa
    domain: switch
mode: single

I wait 15 minutes and just to be sure I make sure the temperature of the extruder has gone below 50 degrees

Finally show the countdown for that 15 minutes:

alias: 3DP CountDown for Shutdown
description: ""
triggers:
  - trigger: state
    entity_id:
      - sensor.k2plus_b4f8_current_print_state
    to: complete
    for:
      hours: 0
      minutes: 0
      seconds: 10
conditions: []
actions:
  - target:
      entity_id: light.wled_2
    data:
      rgb_color:
        - 0
        - 255
        - 0
      effect: Percent
    action: light.turn_on
  - target:
      entity_id: number.wled_intensitat_2
    data:
      value: 100
    action: number.set_value
  - repeat:
      count: 60
      sequence:
        - delay:
            hours: 0
            minutes: 0
            seconds: 15
            milliseconds: 0
        - target:
            entity_id: number.wled_intensitat_2
          data:
            value: >
              {% set calculated_value = 100 - ((repeat.index + 1) * (100/60)) %}
              {% if calculated_value < 0 %}
                0
              {% else %}
                {{ calculated_value | round(1) }}
              {% endif %}
          action: number.set_value
mode: single

That’s it. These are my automations for my 3D printer status

Make Videos work with DaVinci Resolve on Linux Mint

dotnet@oles.cloud (Ole Albers) — Sat, 25 Jan 2025 12:00:00 +0100

Installing DaVinci Resolve on Linux Mint takes a few steps. But at the end it simply works after you have done those.

I tested those on Linux Mint but should work on most Ubuntu - based Distros.

I mostly followed the instructions from virtual curiosities and just added a few of my own additions

Downloading DaVinci Resolve

The first step is to download DaVinci resolve from the official website. You can chose between the free version and the paid version. I really think the paid version is worth the price but the free version nearly allows anything you will need. And of course for testing it should be the first way to go.

After you have downloaded the zip, extract it and open the console on that folder. Make sure the .run- file can be executed:

chmod +x Davinci_Resolve_[version]_Linux.run (replace version with the version you downloaded. There should be only one .run - file) now run the installer: ./Davinci_Resolve_[version]_Linux.run You will get error messages about missing dependencies. Don’t worry. This is to be expected. Use apt install to install all the missing dependencies that are thrown at you e.g.

sudo apt install libasound2 libglib2.0-0 There will be way more dependencies missing. After you have done that restart the installer. Now the next bunch of missing dependencies are shown. Repeat that process until no new dependencies are shown. And yes: This will take up to ten times or so. This does not mean you are in an endless loop. At some point this will stop.

You will most likely find a missing dependency for libasound2 which cannot be installed on Mint. Use libasound2t64 instead.

Once the installer only shows missing dependencies you already installed you can disable the dependencycheck: export SKIP_PACKAGE_CHECK=1 and rerun the installer

Now finally the Wizard should show up and DaVinci gets installed. It will take a while until this is finished.

After the installation you should create a shellscript to run DaVinci (the icon on the desktop created does not work from start): env LD_PRELOAD=/lib/x86_64-linux-gnu/libglib-2.0.so.0:/lib/x86_64-linux-gnu/libgio-2.0.so.0:/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0:/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 /opt/resolve/bin/resolve %u call the file davinci.sh and make sure it can be run: chmod +x ./davinci.sh Now you should be able to start DaVinci with ./davinci.sh

Once you are in DaVinci Resolve you will notice that the first Videos you try to add as media will not display. And the few that do most likely will not have audio. The reason is that on Linux DaVinci is quite picky what Video formats to accept.

Use Piero for the rescue.

This tool automatically created Videos that can be read by Resolve and also creates smaller Proxy files. You will need to install dotnet 8 runtime for this to work

And that`s it. You should be able to work with Resolve on Linux, now.

Automatic Blogpost translation with Claude.ai

dotnet@oles.cloud (Ole Albers) — Thu, 16 Jan 2025 12:00:00 +0100

To be completely honest: I do not really like automatic translations. Especially if you are not born in an english-speaking country, translations have been a real pain from the beginning. MSDN was one of those contenders that did really, really bad translations, years before AI was something to talk about.

Now with AI the translations get better, but there are still bad ones and too many contents get translated. YouTube is nearly not usable anymore from germany if you are interested in original content.

What way to go?

Having said that I still think a translation can be something really useful. First of all it has to be good. And second requirement is that users have a choice.

So I decided to add translations to my blogpost. I stopped blogging in two languages in 2019, so all german entries have been very old. Time to check if there are good solutions out there.

Because I translated a book I already had some comparisons. At least from german to english currently my favorite tool was Claude.io. I did not really like the quality of DeepL and ChatGPT had good results but was very slow. In addition: Claude really seems to like MarkDown. Which is helpful in this case.

This time I wanted to use the API to automatically translate my Grav-Blogitems. Bad news is: You need a paid account for that. Good news is: It does not really matter how many money you use to fill your wallet and it is quite cheap.

My 50KB of data made up about 20 Cents. Of course your milage may vary. This is not sponsored, BTW.

Preparations

After creating an account, you can use the NuGet-Package Anthrophic.SDK. You simply create a new Client with your secret and can start with some messages. If you ever worked with OpenAi-API or Llama creating the message is nearly the same:

  var client = new AnthropicClient(new APIAuthentication(_apiSecret));
var messages = new List<Message>()
{
   new Message(RoleType.User, "Bitte übersetze das folgende Markdown nach Deutsch.");
   
),
new Message(RoleType.User, markdown)
};

In case you are curious the message simply tells to translate a markdown into german

Then you add the parameters. Temperature is - as known for AI-APIS how “creative” the AI is allowed to get. MaxTokens is a required value and can’t be too high:

        var pars = new MessageParameters()
        {
            Messages = messages,
            MaxTokens = 1024,
            Model = AnthropicModels.Claude35Sonnet,
            Stream = false,
            Temperature = 1m,
        };
`Getting the response now is easy:`
var response = await client.Messages.GetClaudeMessageAsync(pars);

What I don’t like: If anything is wrong with your account you do not really receive a useful error code but simply null I had this problem before putting some money into the account.

The response is now in response.Message. The result is great, but long texts just stop after a while. Response.StopReason tells us why:

max_tokens

The obvious solution would be to increase the allowed tokens but as I said MaxTokens has a limited range. So you have to catch that error and go into a loop. Fun thing is: Claude “forgets” what he said before. So you have to tell him and ask to continue afterwards:

 while (response.StopReason == "max_tokens")
 {
   pars.Messages.Add(new Message(RoleType.Assistant, response.Message));
   pars.Messages.Add(new Message(RoleType.User, "bitte fahre fort"));
   response = await client.Messages.GetClaudeMessageAsync(pars);
}

That`s it. Not really that complicated after all.

AI doing AI things

While the result is quite good I had a few issues. Being a coder it is a bit of a learning curve that errors cannot be fixed through coding but only by messages. The first was that the translation is quite formal. An additional message changed that: pars.Messages.Add(new Message(RoleType.User, "Enthält der Text eine persönliche Anrede, verwende 'Du' und nicht 'Sie'. "); (I will not be able to translate this command. It is a really german thing :) )

Another issue is that Claude likes to explain things. So we have to tell him to stop doing that:

pars.Messages.Add(new Message(RoleType.User, "Antworte ausschließlich mit dem Dokument, ergänze keine Erklärungen. "); (Which translates into ‘Only reply with the document, do not explain anything’)

Surprisingly I had to add no additional message to not translate code. If you translate apt-get install into german apt-get installiere for example that would not work. I saw this happen on other sites but Claude did not need any additional hints.

On the other hand. Claude sometimes decided that my code wasn’t interesting enough:

[Rest des Dokuments enthält die gleichen Code-Blöcke und technischen Anweisungen wie im Original, nur der erklärende Text wurde ins Deutsche übersetzt]

Which roughly translates into “the rest of the document contains the same code as the original. Did not translate that”. Did not expect that, but that was fixable as well: pars.Messages.Add(new Message(RoleType.User, "Fasse keine Inhalte zusammen, sondern gebe alles aus. Ziel soll sein dass man das deutschsprachige Dokument genauso alleine versteht wie das englischsprachige "): which roughly translates into “Do not summarize contents. The goal should be that the german document can stand on its own”

Now the result is exactly as needed. For completionists sake here is the complete code:

private async Task<string> TranslateSingleFile(string source)
{
   var translated=string.Empty;
   var markdown = File.ReadAllText(source);
   var client = new AnthropicClient(new APIAuthentication(_apiSecret));
   var messages = new List<Message>()
   {
      new Message(RoleType.User, "Bitte übersetze das folgende Markdown nach Deutsch."
      +" Antworte ausschließlich mit dem Dokument, ergänze keine Erklärungen. "
      + "Enthält der Text eine persönliche Anrede, verwende 'Du' und nicht 'Sie'. "
      +"Fasse keine Inhalte zusammen, sondern gebe alles aus. Ziel soll sein dass man das deutschsprachige Dokument genauso alleine versteht wie das englischsprachige "            
       ),
       new Message(RoleType.User, markdown)
   };

   var pars = new MessageParameters()
   {
      Messages = messages,
      MaxTokens = 1024,
      Model = AnthropicModels.Claude35Sonnet,
      Stream = false,
      Temperature = 1m,
   };

   var response = await client.Messages.GetClaudeMessageAsync(pars);
   if (response?.Message == null) throw new Exception("No valid response from Claude");
        
   translated += response.Message.ToString();
   while (response.StopReason == "max_tokens")
   {      
      pars.Messages.Add(new Message(RoleType.Assistant, response.Message));
      pars.Messages.Add(new Message(RoleType.User, "bitte fahre fort"));
      response = await client.Messages.GetClaudeMessageAsync(pars);     
      translated += response.Message.ToString();
   }

   if (response.StopReason != "end_turn") throw new Exception($"Translation failed. Stop reason:{response.StopReason}");
   return translated;
}

Setting single Panels on NanoLeaf using Home Assistant

dotnet@oles.cloud (Ole Albers) — Sat, 20 Jan 2024 12:00:00 +0100

I went deep into Home Assistant automation currently and - as everybody knows who ever went this road - more ideas came up. So my newest idea was to use the Hex-Panels from Nanoleaf to display some information. For example if the washing machine is ready, I have an appointment, the Smartphone Battery gets low and so on.

This entry describes how I managed to do that.

I collected all this information from different sources. THe most important was from sygys who posted the Home Assistant configuration: https://community.home-assistant.io/t/nanoleaf-per-tile-light-control-build-in-ha/442720 By the API documentation alone I would not have been able to achieve that.

The hardware

I mostly used Nanoleaf Elements in the wooden design. These cannot display any colors but just lighten up. I like the descent appearance of those. Do not want to create an all-rgb-gaming-experience in my living room. I also added three dark black hexagon tiles to it. Those where meant to display the status if needed. BTW: The nanoleaf app does not offer any color scenes in this setup because the wooden panels are the “main” panels. But that does not matter for this scenario.

Integrating in Home Assistant

For the main features I used the standard integration for Home Assistant. That allows detecing swipe controls (another advantage of the wooden panels because you don’t see fingerprints there) and start mostly anything what the app allows. Which does NOT include adressing single panels

Using the API

Currently this can only be done by a REST-Api. Initially you need any tool that can send POST - Commands. You can use Curl if you like to work on the shell, or UI-Tools like Postman.

Retrieving a secret

The first manual step is to obtain a secret. This is done exactly like when installing the Home Assistant integration. You just pair the Panels by pressing the power button for a few seconds until the LEDs start blnking/moving. Don’t worry, this will NOT remove any previous pairings. Now while the pairing is active call the following URL:

POST http://192.168.178.65:16021/api/v1/new

(replace the IP by the Nanoleaf IP. Should be visible in your Router for example)

This will return just a simple value which contains your secret token. This token will remain valid even if you unpower the Nanoleaf. Make sure you store this secret somewhere, you will need it.

Retrieve the Configuration

Now it is time to test your secret and retrieve the configuration from your Nanoleaf panels. You even can do this in your browser, or still in Postman/Curl:

http://192.168.178.65:16021/api/v1/thisisthemagicsecretreturnedbefore/

The most interesting part is the layout/positionData-Part in that response:

      "layout": {
            "numPanels": 46,
            "sideLength": 0,
            "positionData": [
                {
                    "panelId": 8172,
                    "x": 301,
                    "y": 747,
                    "o": 360,
                    "shapeType": 15
                },
                {
                    "panelId": 36653,
                    "x": 276,
                    "y": 703,
                    "o": 300,
                    "shapeType": 15
                },
[..]

I honestly was a bit surprised, because I do NOT have 46 panels. Far from it. The simple truth is: THe wooden Panels have 6 LEDs. each of them has their own ID. So having 7 wooden panels that makes 42 Ids, additional 3 Ids for the three black panels and the control buttons is an additional panel (with id 0)

If you do not need to adress all Panels but just a single one, a good hint is to just rotate one of the panels, make that call again and see which panel has a new orientation (“o” in the json)

Now you have to write down all PanelIds. At least those you want to modifiy. Because I mostly want to use the black panels I split the following part in two, but you can also work with one (or even more).

Configure Rest in Home Assistant

Go to your beloved Home Assistant Configuration (configuration.yaml) and add the following entries: {% raw %}

rest_command:
  nanoleaf_wood:
    url: http://192.168.178.65:16021/api/v1/thisisthemagicsecretreturnedbefore/effects
    method: PUT
    payload: >
      { "write" : {"command": "display", "animType": "static", "animData":
      {%- set all = [ 8172,36653,  32365,61100,6889,35368,57498,28763,33051,4570,9112,45913,61802,25003,37099,42,14434,43171,34741,29680,58161,4721,33456,45298,4960,8482,45539,16547,53346,9255,5361,58801,30064,33077,4596,57524,54769,50577,21840,42000,13521,1683] %}
      {%- set ns = namespace(panels=[panels | count | string]) %}
      {%- for panel in panels %}
      {%- set ns.panels = ns.panels + [ '{} 1 {} {} {} 0 20'.format(all[panel.number-1], panel.r, panel.g, panel.b) ] %}
      {%- endfor %}
      "{{ ns.panels | join(' ') }}",
      "loop": false, "palette": [], "colorType": "HSB"}
      }
    content_type: "application/json"
  nanoleaf_black:
    url: http://192.168.178.65:16021/api/v1/thisisthemagicsecretreturnedbefore/effects
    method: PUT
    payload: >
      { "write" : {"command": "display", "animType": "static", "animData":
      {%- set all = [ 52732, 44680, 17782 ] %}      
      {%- set ns = namespace(panels=[panels | count | string]) %}
      {%- for panel in panels %}
      {%- set ns.panels = ns.panels + [ '{} 1 {} {} {} 0 20'.format(all[panel.number-1], panel.r, panel.g, panel.b) ] %}
      {%- endfor %}
      "{{ ns.panels | join(' ') }}",
      "loop": false, "palette": [], "colorType": "HSB"}
      }
    content_type: "application/json"

{% endraw %} Replace the IP with your Nenoleaf - IP and the secret with your secret (before “/effects”). It is up to you if you want one or more different calls. The numbers in the “set all” - part have to be replaced by the panelIds you retrieved earlier.

What comes next is of course testing your config and then restarting Home Assistant.

Automation Examples

Now you can use this in any automation. The number is the number in the array, starting at 1(!) Example for the simple ones:

alias: Nano_BlackTest
description: ""
trigger: []
condition: []
action:
  - service: rest_command.nanoleaf_black
    data:
      panels:
        - number: 1
          r: 255
          g: 0
          b: 0
        - number: 2
          r: 0
          g: 255
          b: 0
        - number: 3
          r: 0
          g: 0
          b: 255
mode: single
`and an example to light up the three wooden panels (remember. Each panel has 6 LEDs):`
alias: Nano_WoodTest
description: ""
trigger: []
condition: []
action:
  - service: rest_command.nanoleaf_wood
    data:
      panels:
        - number: 31
          r: 255
          g: 255
          b: 255
        - number: 32
          r: 255
          g: 255
          b: 255
        - number: 33
          r: 255
          g: 255
          b: 255
        - number: 34
          r: 255
          g: 255
          b: 255
        - number: 35
          r: 255
          g: 255
          b: 255
        - number: 36
          r: 255
          g: 255
          b: 255
        - number: 19
          r: 255
          g: 255
          b: 255
        - number: 20
          r: 255
          g: 255
          b: 255
        - number: 21
          r: 255
          g: 255
          b: 255
        - number: 22
          r: 255
          g: 255
          b: 255
        - number: 23
          r: 255
          g: 255
          b: 255
        - number: 24
          r: 255
          g: 255
          b: 255
        - number: 1
          r: 255
          g: 255
          b: 255
        - number: 2
          r: 255
          g: 255
          b: 255
        - number: 3
          r: 255
          g: 255
          b: 255
        - number: 4
          r: 255
          g: 255
          b: 255
        - number: 5
          r: 255
          g: 255
          b: 255
        - number: 6
          r: 255
          g: 255
          b: 255
mode: single

That’s it. Have fun with your panels-

How to track a squirrel

dotnet@oles.cloud (Ole Albers) — Sun, 17 Dec 2023 12:00:00 +0100

[Disclaimer: This is not meant as a tutorial, more as some chitchat. You can find the source on github nonetheless, but I assume this is a VERY specific use case noone else uses]

Let’s start by telling you that this whole project ist utter nonsense. Noone really needs this feature. I decided to create this nonetheless.

The Goal

I own a little feeding box for squirrels of the neighbourhood. And it works as designed: Squirrels get here, steal their food and run away.

I can see them through my glass door and it just looks amazing, especially in the typical “Hamburger Wetter” (Hamburg weather, means: raining horizontally). Sometimes I take a picture or even a video, but through the door that just does not look very great.

So I decided to add a surveillance cam to the mix, and while I am on it: Automatically publish whenever a squirrel appears

Hardware

I use a dlink - outdoor webcam with WiFi. Not really worth looking up the model number here, just the cheapest WiFi-Cam I could find which is compatible with Synology Surveillance Station (but nearly all are)

In addition the Synology NAS with installed surveillance station. I own a DS920+, but any should do

Privacy

I do not want to track any persons. Not me, not others. This is not meant to be surveillance, it is meant to film squirrels. TBH I did not ask the squirrels how they think about privacy and GDPR, but I assume they are fine with it. After all they accepted all Cookies until now, and not only the required ones.

To make sure only squirrels (or other animals) get caught I wanted to do a few conditions:

Only track movement directly in front of the feeder.
- So I can make sure not Humans or Traffic in the background trigger the motion sensor
Put the Camera in “Squirrel”-height.
- So even if someone (me or my guests) manages to trigger the alarm no faces can be seen
Have a simple to use approval workflow
- Just in case. If someone ends up lying there drunk I do not want to publish a video of that person anywhere

Only Track movement directly in front of the feeder

That is one of the reasons why I use the Surveillance Station of Synology. The Camera itself has movement detection and (according to the documentation) even a fancy AI “this is a person” algortihm. I do not want the last for obvious reasons and especially can not decide which area of the image should be used for detection. In addition this seems to require a cloud feature. I do not want my camera feed in the cloud from anyone. This should never leave my house without my consent. The surveillance station can select very small pieces of the image to be selected as alert regions. I put it directly in front of the feeder where no background movement can be detected. This way I can also adjust the sensibility of the detection to quite a high value because no noise becaues of branches moving in the wind etc. is to be expected.

Put the camera in squirrel-height

That was easy as there are some flowerboxes directly before the feeder where I mounted the cam

Have a simple approval workflow

Because I mostly am at Mastodon at time of this writing (these sentences sometimes age very bad. A few months ago that would have been the artist formerly known as Twitter, which itself was just a replacement for Google+ which, well, you know the sad story). Let’s just hope that was the last Social Media switch for me.

One of the advantages of Mastodon is that it has a very nice, simple API that allows sending toots, getting Notifications and so on. So my idea for the workflow is to notify me (and only me) that a new movement has been detected. Attach an image of that movement and if I approve this (e.g. by adding a “fav” to that toot) the bot knows that this detection was a valid Squirrel video and posts it. So AI without the “A”. (some might argue even without the “I”…)

Synology

Adding the Webcam is quite simple. It nearly is found automatically in Surveillance station. Finding out how to connect was a bit harder. Username is always “admin” and the required password is a PIN. Printed onto the Power Plug, not the device… After that the App of the camera does not like the Camera anymore, which is bonus for me as I do not want images to be uploaded to dlink, anyways.

The detection area can be set in the Camera options. Fiddling with the thresholds takes a while, especially as I do not have a trained squirrel to assist me on that. I then changed the time plan to always motion detection (so disabling the constant recording) and updated the minimum record time to five minutes. A good comprimise between getting everything and not creating videos that are too big for mastodon/youtube/wherever I wanted to upload this, which I had not decided at this point

Synology has multiple ways to notify about movements: Email, SMS, Push notifications and Webhooks. Obviously the last one was the way to go as it does not spam me in any way and I have to program stuff anyways.

Writing a Webhook

A webhook is just an HTTP-call essentially. Synology allows POST and PUT in Surveillance Station. So I created a simple Web Api in Visual Studio and was ready to debug the incoming data.

Firewalls, Firewalls everywhere!

Synology allows you to send a test-notification which just resulted in an error. WHAT exactly went wrong is nowhere to be seen. Not in surveillance station and not in the logs from the synology. So I had to fly blind at this point.

I knew it worked when using my local machine, either via localhost or the ipadress from my development machine.

So the next step was to connect to my synology via SSH and try to access my API throuh a simple webcall via GET: wget https://192.168.178.23:6789/ping

This simple call already did not work. So must have been a firewall issue. I checked everything in my Synology configuration FOR A VERY LONG TIME and did find nothing. Finally I put my attention to Windows, disabled the Defender Firewall and it went through. Because I do not want to have the firewall open I added an exception through the “allow an app through windows firewall” and finally it did work with the firewall still enabled.

Receiving data from the webhook

The Webhook arrived as expected, has some informations in it and can even provide an url to a preview image. That image URL is a generated “public” (within my LAN) Address I am able to receive and download:

app.MapPost("/squirrel", async context =>
{
    var requestBody = await context.Request.ReadFromJsonAsync<SynologyBody>();
    if (requestBody != null)
    {
        if (!string.IsNullOrWhiteSpace(requestBody.Capture))
        {
            var previewImage = await downloader.DownloadFile(requestBody.Capture);                                    
        }
        else
        {
            Console.WriteLine("received event without image. Will ignore it");
        }
    }
    await context.Response.WriteAsync("event received");
});

public class SynologyBody {
  public string Subject { get; set; } 
  public string Desc { get; set; } 
  public string Capture { get; set; } 
}

(Synology does not really care about the response but I found it useful when testing locally through Postman)

At this point I do not have a video yet which is to be expected, as “normal” users would expect to receive the warning immediately and not five minutes after someone stole their nuts.

Send private notification to Mastodon

After an event have been received the Image is downloaded and a message is sent to Mastodon using MastoNet:

private MastodonClient Login()
{
    return new MastodonClient(_instance, _accessToken);
}

public async Task<Status?> SendPreview(Stream previewFile, DateTime notificationDate)
{
    var client = Login();
    var text = $"{_approver} ALARM! Possible Squirrel detection! ALARM! #preview DATE:\n{notificationDate}\n";
    var media=await client.UploadMedia(previewFile, "squirrel.png", "possibly a squirrel");
    var status=await client.PublishStatus(text, Visibility.Direct, mediaIds: new[] {media.Id }); 
    if (string.IsNullOrWhiteSpace(status.Id))
    {
        Console.WriteLine("Could not send preview");
        return null;
    }
    Console.WriteLine($"Published to Mastodon with id {status.Id}");
    return status;
}

This creates the following toot:

Waiting for approval

This toot has only been sent to me privately (mastodon has no encryption so always keep that in mind about the “privacy” - part). I can then decide that this is a valid Squirrel detection by setting the “fav” star of that toot.

Keeping the state

To be able to understand which previewtoot belongs to what webhook call I need to store the state. I uses LitDB for this. LiteDB is a very simple document database which needs no installation at all. It completely only lives within the NuGet-Package and the data is stored in a Bson-Format (Json, but compressed). It is a bit like simply serializing my contents and store it into a Json file. Just performing much better.

That is why the complete “database”-class is as simple as this:

 public class DataBase
    {
        private const string _databaseName = @"Filename=squirrel.db;connection=shared";

        public T GetContents<T>(string id)
        {
            using var db = new LiteDatabase(_databaseName);
            return db.GetCollection<T>().FindById(new BsonValue(id));
        }

        public void RemoveContents<T>(string id)
        {
            using var db = new LiteDatabase(_databaseName);
            db.GetCollection<T>().Delete(new BsonValue(id));
        }

        public void UpsertContents<T>(T contents)
        {
            using var db = new LiteDatabase(_databaseName);
            db.GetCollection<T>().Upsert(contents);
        }

        public SquirrelData? FindByFilename(string filename)
        {
            using var db = new LiteDatabase(_databaseName);
            return db.GetCollection<SquirrelData>().FindOne(q => q.ExpectedMovieFile == filename);
        }
    }

Now the data inside the Database contains the following stuff:

public class SquirrelData
    {
        [BsonId]
        public string Id { get; set; }  // First MastodonID
        public string? PublicId { get; set; }

        public DateTime EventReceived { get; set; }
        public DateTime SentForApproval { get; set; }
        public DateTime? SentPublic { get; set; }
        public DateTime? Faved { get; set; }
        public string? ExpectedMovieFile { get; set; }
    }

The Id simply is the Id from the message just posted to mastodon. This object mostly contains Mastodon-Information and Information about the WebHook event.

As mentioned above the WebHook has no information about the Video that is created (or most likely: Is still being created when the WebHook is sent). So we simply have to find it. The rules are quite simple: Within the directory with surveillance videos it is simply the newest file we can find.

One problem remains: The preview image was within the WebHook and did not need authentication do be downloaded. The surveillance directory does need authentication. So I created a user for that specific purpouse allowed it to access the folder and needed to do some SMB-Stuff for authentication.

I used the SMBLibrary for that. It isn’t as straight forward as navigating through a filesystem, but it does work fine. First I have to connect using my credentials:

 public void Connect()
{
    var smbClient = new SMB1Client(); 
    bool isConnected = smbClient.Connect(IPAddress.Parse(_server), SMBTransportType.DirectTCPTransport);
    if (isConnected)
    {
        NTStatus status = smbClient.Login(string.Empty, _username, _password);
        if (status == NTStatus.STATUS_SUCCESS)
        {
            _smbClient=smbClient;
            return;
        }
    }
    throw new Exception("Connecting failed. Check your credentials");
}

After successfully connecting I can access the share and search for the most current mp4-file recursively:

     
public string FindMostCurrentVideo()
{
    var store = _smbClient.TreeConnect("surveillance", out var status);
    if (status != NTStatus.STATUS_SUCCESS) throw new Exception("Connecting to Share failed");
    var mostCurrentFile=GetMostCurrentFileRec((SMB1FileStore)store, _startPath);
    return Path.Combine(mostCurrentFile.Path, mostCurrentFile.Filename);
}

private MostCurrentFile GetMostCurrentFileRec(SMB1FileStore store, string folder)
{
    folder += "\\";
    var mostCurrentFile = new MostCurrentFile { Created = DateTime.MinValue };

    var status=store.QueryDirectory(out var subdirectoryList, folder, SMBLibrary.SMB1.FindInformationLevel.SMB_FIND_FILE_DIRECTORY_INFO);
    if (status== NTStatus.STATUS_SUCCESS)
    {
        object directoryHandle;
        FileStatus fileStatus;
        status = store.CreateFile(out directoryHandle, out fileStatus, folder, AccessMask.GENERIC_READ, SMBLibrary.FileAttributes.Directory | SMBLibrary.FileAttributes.Normal, ShareAccess.Read | ShareAccess.Write, CreateDisposition.FILE_OPEN, CreateOptions.FILE_DIRECTORY_FILE, null); ;
        if (status == NTStatus.STATUS_SUCCESS)
        {
            List<FindInformation> fileList2;
            status = store.QueryDirectory(out fileList2, folder+ "*", FindInformationLevel.SMB_FIND_FILE_DIRECTORY_INFO);
            var dirInfo = fileList2.Where(q => q.GetType() == typeof(FindFileDirectoryInfo)).Cast<FindFileDirectoryInfo>();
            var subdirectories=dirInfo.Where(q=>q.ExtFileAttributes.HasFlag(ExtendedFileAttributes.Directory));
            var files=dirInfo.Where(q => !q.ExtFileAttributes.HasFlag(ExtendedFileAttributes.Directory));
            // Look for subdirectories and drill down to them:
            foreach (var directory in subdirectories)
            {
                if (directory.FileName.Contains(".")) continue;
                var mostCurrentSubFile = GetMostCurrentFileRec(store, Path.Combine(folder, directory.FileName));
                if (mostCurrentSubFile.Created > mostCurrentFile.Created) mostCurrentFile = mostCurrentSubFile;
            }

            // Look for .mp4 - Files in current directory and check if they are newer than the most current file known:
            foreach (var file in files.Where(q=>q.FileName.EndsWith(".mp4")))
            {
                if (file.CreationTime > mostCurrentFile.Created)
                {
                    mostCurrentFile = new MostCurrentFile
                    {
                        Created = file.CreationTime.Value,
                        Filename = file.FileName,
                        Path = folder
                    };
                }
            }
            status = store.CloseFile(directoryHandle);
        }
    }
    return mostCurrentFile;
}

At this point I do not need the filecontents itself, but only the path so I can use it lateron.

After that we have the whole WebHook complete:

Reveive the Contents (including preview image) from the webhook
Determine which Video File matches
CHeck if we already had this file (if within a video multiple webhooks are fired) and avoid duplicates
Send a notification to Mastodon
Store all known information into our database.

So our Webhook from the beginning now looks like this:

app.MapPost("/squirrel", async context =>
{
  var received = DateTime.Now;
  Console.WriteLine("Received squirrel Event");
  var requestBody = await context.Request.ReadFromJsonAsync<SynologyBody>();
  if (requestBody != null)
  {
    if (!string.IsNullOrWhiteSpace(requestBody.Capture))
    {
      var previewImage = await downloader.DownloadFile(requestBody.Capture);
      downloader.Connect();
      string videoFilename = downloader.FindMostCurrentVideo() ?? throw new Exception("No Video Found");
      var existingItem = database.FindByFilename(videoFilename);
      if (existingItem != null)
      {
        Console.WriteLine($"'{videoFilename}' already processed with MastodonId {existingItem.Id}");
        await context.Response.WriteAsync("duplicate event");
        return;
      }
      var mastodonStatus = await mastodon.SendPreview(previewImage, received);
      if (mastodonStatus != null)
      {
        database.UpsertContents(new SquirrelData { EventReceived = received, ExpectedMovieFile = videoFilename, Id = mastodonStatus.Id, SentForApproval = mastodonStatus.CreatedAt });
      }
    }
  }
  await context.Response.WriteAsync("event received");
});

Checking for Notifications

As mentioned the “approval workflow” (a big word for this simple mechanism, I know) is to wait for a fav. If the fav is received it means I decided this is a valid Squirrel detection and the video can be published.

Also do not wait longer than 7 days. If I did not approve in that time, it probably is not a good video.

Again using MastoNet I can receive the Notifications for that user:

public async Task<List<Notification>> GetFavs()
{
  var favourites= new List<Notification>();
  var client=Login();
  var notifications=await client.GetNotifications();
  foreach (var notification in notifications)
  {
    switch (notification.Type)
    {
      case "mention":
        await client.PublishStatus($"Sorry. I am just a bot and will not respond to this. Ask {_approver} if you want to know anything about what I do", replyStatusId: notification.Id);
        await client.DismissNotification(notification.Id);
        break;
      case "favourite":
        favourites.Add(notification);
        break;
      default:
        Console.WriteLine($"discarding notification of type {notification.Type}");
        await client.DismissNotification(notification.Id);
        break;
    }
  }
  return favourites;
}

The nofication.Status.Id now is the id we already stored in the Database and can use to find all known WebHook-Data of that event and send a new Toot. This time just with the Video file instead of the preview image.

Why no Mastodon Webhooks?

The Mastodon API also offers Webhooks. So instead of periodially polling for new notifications this could be used to receive the favs. The main reason why I did not use this (apart from the fact that polling is simply fast enough) is the fact that my application is not accessible from the outside. To allow this I would need to make my NAS (where the application is hosted) accessible from the Internet. And I do not want that. I could add Firewall rules to only allow specific ports and so on, but no access is even a bit better than a limited access. Even if noone would get through, just bots trying the flood my machine with requests is a headache I do not need.

Uploading the Video

After trying to uplaad the Video directly to Mastodon the next problem came up: The file was too big. While it was way inside the limits the upload (to the mastodon.social instance) just responded with an error “something went wrong”. I suspect the Library to be the main issue as the Upload is not done async and so a timeout might be the cause.

That is why I decided to upload the Video to YouTube instead. YouTube provides APIs to do that. The main issue is that this also requires an OAuth authentication. Luckily the documentation for this are quite straight-forward so a simple Test worked fine:

public async Task UploadVideo(Stream fileStream)
{
  var youtubeService = new YouTubeService(new BaseClientService.Initializer()
  {
      ApiKey = _apiKey,
      ApplicationName = this.GetType().ToString()
  });
  
  UserCredential credential;
  using (var stream = new FileStream("secrets.json", FileMode.Open, FileAccess.Read))
  {
    credential = await GoogleWebAuthorizationBroker.AuthorizeAsync(GoogleClientSecrets.Load(stream).Secrets,new[] { YouTubeService.Scope.YoutubeUpload },"user",CancellationToken.None);
  }

  youtubeService = new YouTubeService(new BaseClientService.Initializer()
  {
      HttpClientInitializer = credential,
      ApplicationName = Assembly.GetExecutingAssembly().GetName().Name
  });

  var video = new Video();
  video.Snippet = new VideoSnippet();
  video.Snippet.Title = "Squirrel detected";
  video.Snippet.Description = "You should have bought a squirrel";
  video.Snippet.Tags = new string[] { "squirrel" };
  video.Snippet.CategoryId = "22"; 
  video.Status = new VideoStatus
  {
      MadeForKids = false,
      SelfDeclaredMadeForKids = false
  };
  video.Status.PrivacyStatus = "private"; 
  var videosInsertRequest = youtubeService.Videos.Insert(video, "snippet,status", fileStream, "video/*");
  videosInsertRequest.ProgressChanged += videosInsertRequest_ProgressChanged;
  videosInsertRequest.ResponseReceived += videosInsertRequest_ResponseReceived;

  await videosInsertRequest.UploadAsync();
}
private void videosInsertRequest_ResponseReceived(Video obj)
{
  Console.WriteLine($"done");
}

private void videosInsertRequest_ProgressChanged(IUploadProgress obj)
{
  Console.WriteLine($"Bytes: {obj.BytesSent}| Status: {obj.Status}");
}

(secrets.json contains a client_id and client_secret for oauth) While uploading the ProgressChanged event gets fired a few times. Obj.Status cycles from Starting, Uploading to Completed. Which is also the Status when ResponseReceived is fired.

Because I had to switch to YouTube I also changed the workflow a bit. Now I already upload the video when posting the preview toot as private on YouTube. So I can verify the complete contents before an approval. And YouTube can process the video while it is waiting for approval. At the end I just switch from private to unlisted if it is approved or delete the Video if no approval has been made for a period of time.

Another Pitfall

YouTube does not allow using service accounts when uploading Videos. This was crucial for my application because this should run on a server. While the code above works I hit a dead end here. So back to Mastodon:

FfMpeg

To reduce the filesize I went to FFMpeg. I can compress Videos and - if required - even resize them. Because I do not want to call something on the console, I needed a wrapper. I found FfmpegCore which does exactly what I want:

public async Task<bool> ResizeVideo(string source, string target)
{
	return await FFMpegArguments
		.FromFileInput(source)
		.OutputToFile(target, true,
		options=>options
			.WithVideoCodec(VideoCodec.LibX265)
			.WithVideoFilters(filter=>filter
			.Scale(videoSize: FFMpegCore.Enums.VideoSize.FullHd)
			)
		.WithFastStart())
		.ProcessAsynchronously();
}

This tiny little piece of code takes any videofile, compresses it as h265 (using default quality settings) and resized it to FullHD (1080 height) maintaining the correvt aspect ratio.

That all works fine on my machine, but because I want to host this (like all my bots) on my Synology-NAS I had to dockerize the contents. I was in luck this time, but this also turned out to be quite easy:

RUN apt-get -y update && apt-get -y upgrade && apt-get install -y --no-install-recommends ffmpeg A single Line in the DockerFile and everything was working smoothly

Fixing the Api

The reduced Filesize did not solve the problem that Mastodon did not want to upload my file. So I digged a bit deeper into the Mastodon API. The media upload used by MastoNet seemed to marked as depricated (v1) exactly for the reason why I did not work in my project: It was meant for small preview images, not for Video files.

A V2-Version is available but not implemented by MastoNet. So I forked the packages from Github and added a V2-Version:

public async Task<Attachment> UploadMediaAsync(MediaDefinition media, string? description = null,
AttachmentFocusData? focus = null)
{
	media.ParamName = "file";
	var list = new List<MediaDefinition>() { media };
	var data = new Dictionary<string, string>();
	if (description != null)
	{
		data.Add("description", description);
	}

	if (focus != null)
	{
		data.Add("focus", $"{focus.X},{focus.Y}");
	}

	return await this.Post<Attachment>("/api/v2/media", data, list);
}

This was confusingly easy as I simpy just replaced “v1” by “v2”. Still it worked. Problem was: The next step, usually taking the returned Id and add this to a toot failed, because Mastodon was not finished, yet. I could simply have waited a long enough time, but decided to add a poll to receive status updates. Every Media of a known I can be requested. There is no statusflag about it processing but some properties give a hint: First there is the previewUrl this is filled at the very start after uploading. Tbe Url-Property, however isn’t. This will always be Null until everything is processed. So just requesting the Media every 10 seconds until the Url is filled did the trick;

public async Task<Attachment> WaitUntilMediaIsUploaded(Attachment attachment, int waitSeconds = 10, int maxWaitSeconds = 300)
{
	int totalTimeWasted = 0;
	while (attachment.Id != null && attachment.Url == null && attachment.PreviewUrl != null && totalTimeWasted < maxWaitSeconds)
	{
		Thread.Sleep(waitSeconds * 1000);
		totalTimeWasted += waitSeconds;
		attachment = await GetMedia(attachment.Id);
	}
	return attachment;
}

public async Task<Attachment> GetMedia(string mediaId)
{
	var attachment = await this.Get<Attachment>("/api/v1/media/" + mediaId);
	var prev = attachment.PreviewUrl;
	var url = attachment.Url;
	return attachment;
}

Finally

So that was the last Pitfall. Now everything works:

If a motion is detected my WebHook gets called
The Webhook posts a previewimage as a direct toot (only be seen by me) to Mastodon including a Preview Image
If “fav” the toot when this is a valid Squirrel detection
The Bot checks for new Favs and finds mine
The according Video is found in the LiteDB
The Video gets downloaded via SMB-Stuff
The Video gets Resized by FfMpeg
The resized video gets uploaded to Mastodon
A public toot containing the video is created

(Using standard Mastodon-Features, all toots will get deleted after two weeks reducing the space required)

But WHY?

I am very aware that this was mostly a waste of time. A lot of time for a simple “Lets show when there is a squirrel”. Simone Giertz once did a great TED - Talk Why you should make useless things. While she talks about robots, by fun thing to do was alway coding. No matter how useless the results are. So: Yes. I totally agree with her. Creating stupid things just because it is fun, with noone asking why this takes so long and even be able to quit if something becomes boring is a nice diversion to the normal coding reality I have for paying the rent.

I hope you also had some fun reading this; I had writing it :)

Raspberry Pi 4: usB first!

dotnet@oles.cloud (Ole Albers) — Wed, 27 May 2020 12:00:00 +0200

At the time of this writing the boot-from-usb has just been made available as a Beta feature for the Raspberry Pi 4.

There are a few tutorials on how to activate that feature but I had an additional requirement: Boot from USB first, then boot from SD which requires a few additional steps.

This document might still be useful if you just want the SD first approach. Just ignore the second last part in that case.

But before we begin:

Use at your own risk. We are writing data to the EEPROM here using beta contents and even manipulate configs for that. If something fails your pi might be bricked. Not even reinstall Linux will repair that.

Additional warning: Don’t try this on a N00bs install.

Still here? So let’s begin. (I will asume you use Raspbian Buster)

Updating the Pi

Before we start we should always have the most current Linux version installed. So make sure your internet connection works and enter

sudo apt-get update

followed by

sudo apt-get upgrade

and in addition

sudo rpi-update

to update the firmware on the sdcard

Switching to beta

now change rpi to switch to beta - state. (If you are reading this from the future: Hi there! Cars still a thing? And maybe you don’t need to switch to beta anymore)

do this by using your favourite editor like vi or nano to change the rpi-config:

sudo nano /etc/default/rpi-eeprom-update

change the release-status from critical to beta and save the file.

Flashing the EEPROM

Now check the most current version on your pi:

ls /lib/firmware/raspberrypi/bootloader/beta/pieeprom*.bin

At time of this writing the most current version there is 2020-05-15. So flash the eeprom with that version:

sudo rpi-eeprom-update -d -f /lib/firmware/raspberrypi/bootloader/beta/pieeprom-2020-05-15.bin

You will be thrown back to the prompt a while after. Now reboot your pi and hope nothing went wrong:

sudo shutdown - r now

now check if the version has been installed by typing

sudo vcgencmd bootloader_version

the response should display the date corresponding to your file. So something like May 15

Changing the boot order

By default now the boot order will be SD first, USB second. So if you just want to throw away your SD and boot grom USB: Ignore this chapter. If you need a “hardware multiboot” (boot from USB if attached but still have a fallback SD in place if not) follow these instructions:

First make sure I did not lie about the default order and display the current config:

sudo vcgencmd bootloader_config

this should display some information about the configuration. Most interesting for us is the last part called BOOT_ORDERwhich will contain the entry

BOOT_ORDER=0xf41

where 4=USB and 1=SD. More details like network boot can be found at the bootloader documentation.

While this might look like just how we want it you have to read this right to left. So it is SD (1) first and USB (4) second. But we are about to change that.

First it is highly recommend to not fiddle with the official ROM files but make a copy of that. Let’s do that:

sudo cp /lib/firmware/respberrypi/bootloader/beta/pieeprom-2015-05-15.bin pieeprom.bin

Now create a config from that rom:

sudo rpi-eeprom-config pieeprom-usbfirst.bin >bootconf.txt

You now created a config file containing the config-output which you can edit by nano again:

sudo nano bootconf.txt

and change the last line to:

BOOT_ORDER=0xf14

At the last step we now apply our changed config und update the eeprom again:

sudo rpi-eeprom-config --out pieeprom-usbfirst.bin --config bootconf.txt pieeprom.bin sudo rpi-eeprom-update -d -f ./pieeprom-usbfirst.bin sudo shutdown -r now

Finalizing the USB drive

Now our raspberry can boot from USB (and boot from there first) but the linux on our USB does not really know how to handle that. This part is simple:

First you need to flash a Raspbian image onto your USB-drive (just like you would to with an SD-card) . Then attach and mount the usb drive and copy the most current firmware files from the /boot - directory of the sdcard to the Usb drive. Alternatively you can select those files from the master-branch on github.

Have fun :)

FlashAirDownloader - development: Now it's your turn

dotnet@oles.cloud (Ole Albers) — Tue, 03 Jan 2017 12:00:00 +0100

Hi,. folks,

I just programmed a small windows service to automatically FlashAir-pictures when I found out that I lost my FlashAir - card

So I am not able to test that program anymore. I decided for myself that I need something better than the unstable FlashAir-Cards and bought a camera with WiFi inside. (Just in case you are curious: It is a Sony HX400)

I do not publish this as a working download, because I am not able to test it. So there is a big chance that this even breaks the original program. But FAD2 is open source, so I checked the code in into the development-branch on GitHub.

So if there is any C# - developer out there who is willing to invest one or two hours to finish that, feel free to do that.

P.S: No, please don’t buy me a new FlashAir :)

Two Apps at the same time: Deploy SharePoint Add-In to Cloud AND On-Premise

dotnet@oles.cloud (Ole Albers) — Tue, 13 Dec 2016 12:00:00 +0100

What the fuzz is this about?

I am not absolutely sure if I am the first one who did this, but it seems at least noone else wrote it down until now. :) We hat a customer who liked our Office365 Provider-Hosted App “Mydea”, but wanted us to install that on his SharePoint 2013 - Installation. In Germany a lot of companies do not trust the cloud. So we had three options:

Tell the customer this does not work
Copy&paste the whole solution and change it so it works on OnPrem
Do something else

The first option - of course - wasn’t valid for me. The second would have worked. But I did not want to end up in maintanance hell. Our product is about innovation management. And the most innovative about that is the product itself. We develop agile, adding new features (and sometimes bugs) continuously. We do not want to do that twice.

So I ended up on the third option: Changing my solution so it will run in an Office365 - SharePoint and on an On-Premise installation of SharePoint 2013 (or above).

Common sense about that path seems to be: That does not work. But it does. It just is a real bumby road to get there. This also means: You should know the basics. You should have created at least one Add-In for Office365 and one for SharePoint 2013 or above. I will not explain these steps here (which are complicated enough for local installation). I also cannot help you there in the comments or by mail.

There are plenty of documents out there that explain what’s needed for this. You should read those, first.

If you find this article useful and earn lots of money because of it: Feel free to buy me a beer :)

Analyzing the differences: What has to be done?

I started with an Office365-App and added the possibility to run that App under an OnPrem-Installation, too. But you should be able to go the other way with little modifications.

My first step was to create two dummy “Hello World"ish Applications. One for O365 and one for an OnPrem-Installation and look at the source, to “diff” it. These are the conclusions I came around:

At most of us use NuGet-Packages, that was the starting point. There are different Packages: You need the following:

OnPrem: - AppForSharePointWebToolKit

Office 365: - AppForSharePointOnlineWebToolKit - Microsoft.SharePointOnline.CSOM

These Packages result in different DLLs with the Prefix “Microsoft.SharePoint.Client.*”, but different Versions. OnPrem: - Version 15.xxx Office 365: - Version 16.xxx

The Sourcecode of the *.cs - files created by that NuGet-Packages (_SharePointContext.cs_ & _Tokenhelper.cs_) are exactly the same. So no need to change anything there.

Finally you usually store your Office365-Apps on Azure, but don’t for On-Premise-Installations. This results in different Authentications: OnPrem: - Using Certificates - Using Windows Authentication

Office 365: - Using ClientSecret - Using OAuth Authentication

These Informations are stored inside the Web.Config

Before we start

Create a new Branch in your repository. You never know what can go wrong :)

The Web-Project (MVC most likely)

Creating different Build configurations

The main purpose if this action was to have just one CodeBase for both versions. So new features or bugfixes have only to be made once. No extra code, no branch, no nothing.

So I started by creating new Build Configurations. You can create them using “Build - Configuration Manager”:

Create two Configurations. One for Debug and one for Release. Make sure to copy the Settings from the existing “Debug”- and “Release”-Configuration

At this point, only do this for your web project and - if you have - any backend-projects that are used by your webproject. We will add others later on, but this way you have the chance to validate every single step:

Now its time to create your config file. Find your web.config, click on it with the right mouse button and select “Add config transform”. After that you should have a Config file for each configuration

Nuget-References

I used the PnP - Packages to develop our O365-Add-In. If you only use the “base” Packages that come with the Add-In-Solution-Template you might have different packages. Nonetheless you will always have a package “AppForSharePointOnlineWebToolKit”.

Package-wise the main difference between the O365/OnPrem - road are the Azure- and OAuth-packages that are only on the 365-PnP. Also the Office365 Libraries use Version 16, While OnPrem use Version 15.

To be able to keep your application updated no matter what configuration you use, create two dummy projects e.g. dummy365 & dummyOnprem. It does not matter what kind of projects these are. The only purpose of these projects is that there are always the packages for both O365 and OnPrem in the ./packages - folder.

Add the O365-pnp-NuGet-packages to your dummy365 and the 2013-pnp-nuget-package to your dummyOnPrem

Now: Before you continue, **make a backup of you .csproj - file. It conains information that will be lost after the next step.

Now add the “other” PnP-Package. If you already have the SharePointPnPCoreOnline-Package installed. now install the SharePointPnpCore2013 NuGet-Package (and vice-versa)

csProj-Changes

You can now diff your csproj - files before and after. Or simply believe my words: There are now two differences. One of them is the OfficeDevPnP.Core - package. The HintPath from the O365 contains SharePointPnPCoreOnline, the HintPath from the 2013-Version contains SharePointPnpCore2013. This does not seem to be an issue as both DLLs are identical. Nonetheless we will have a look at it. Just in case.

The more interesting detail is the Reference to “Microsoft.Online.SharePoint.Client.Tenant”. The Office365 - Version uses Version 16.x and the OnPrem - Version uses 15.x. This is absolutely something we have to care about.

Luckily, the csProj allows some conditions. So create the following new lines:

<ItemGroup Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">
<Reference Include="Microsoft.Online.SharePoint.Client.Tenant, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c, processorArchitecture=MSIL">
<HintPath>..\\packages\\SharePointPnPCore2013.2.9.1611.0\\lib\\net45\\Microsoft.Online.SharePoint.Client.Tenant.dll</HintPath>
<Private>True</Private>
</Reference>
<Reference Include="OfficeDevPnP.Core, Version=2.9.1611.0, Culture=neutral, PublicKeyToken=3751622786b357c2, processorArchitecture=MSIL">
<HintPath>..\\packages\\SharePointPnPCore2013.2.9.1611.0\\lib\\net45\\OfficeDevPnP.Core.dll</HintPath>
<Private>True</Private>
</Reference>
</ItemGroup>

<ItemGroup Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">
<Reference Include="Microsoft.Online.SharePoint.Client.Tenant, Version=16.1.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c, processorArchitecture=MSIL">
<HintPath>..\\packages\\Microsoft.SharePointOnline.CSOM.16.1.5813.1200\\lib\\net45\\Microsoft.Online.SharePoint.Client.Tenant.dll</HintPath>
<Private>True</Private>
</Reference>

<Reference Include="OfficeDevPnP.Core, Version=2.9.1611.0, Culture=neutral, PublicKeyToken=3751622786b357c2, processorArchitecture=MSIL">
<HintPath>..\\packages\\SharePointPnPCoreOnline.2.9.1611.0\\lib\\net45\\OfficeDevPnP.Core.dll</HintPath>
<Private>True</Private>
</Reference>
</ItemGroup>

Do not just copy/paste the content from here! Use the lines from within your csProj-File instead! Versions will change over time. And when you read these lines, my version used might already be outdated.

As a last step, remove the original lines from within the main ItemGroup containing these two libraries.

Well done. You should now try to rebuild your solution on all your four Build configurations, where the OnPrem-Configurations should create a Microsoft.Online.SharePoint.Client.Tenant.dll with Version 15 in your bin-directory and the Office365-Version should create a V16-dll. If you encounter a problem try to unload and reload the projects.

Please be advised: I am not sure what happens if you update one of those two Nuget-Packages. You might have to modify your csproj-file again

Web.Config

Now its time to modify your Web.Configs.

Add the following lines just before the closing "” section into your *.OnPrem.config - files:

<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly xdt:Transform="Replace" xdt:Locator="Condition(./\_defaultNamespace:assemblyIdentity/@name='Microsoft.Online.SharePoint.Client.Tenant')">
<assemblyIdentity name="Microsoft.Online.SharePoint.Client.Tenant" publicKeyToken="71e9bce111e9429c" culture="neutral" />
<bindingRedirect oldVersion="0.0.0.0-15.0.0.0" newVersion="15.0.0.0" />
</dependentAssembly>
</assemblyBinding>
</runtime>

As before: If at time of reading, the most current version is not 15.0, but 15.1 or higher replace the version numbers instead.

Now it is time to modify your “normal” Configs for Debug and Release

<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly xdt:Transform="Replace" xdt:Locator="Condition(./\_defaultNamespace:assemblyIdentity/@name='Microsoft.Online.SharePoint.Client.Tenant')">
<assemblyIdentity name="Microsoft.Online.SharePoint.Client.Tenant" publicKeyToken="71e9bce111e9429c" culture="neutral" />
<bindingRedirect oldVersion="0.0.0.0-16.1.0.0" newVersion="16.1.0.0" />
</dependentAssembly>
</assemblyBinding>
</runtime>

This is not 100% necessary at this time as these values should already be in the main Web.Config, but this makes sure it still works if that changes. Again: Check the version number.

Authentication

Your O365 - Config Files should still contain something like this:

<appSettings>
<add key="ClientValidationEnabled" value="true" />
<add key="UnobtrusiveJavaScriptEnabled" value="true" />
<add key="ClientId" value="aaaaaaaa-beef-face-bbbb-cchcchcchcch" />
<add key="ClientSecret" value="lsdschnabbeldiwuppsfkjashriuae=" />
</appSettings>
`now your OnPrem-configs looks a little bit different:`
<appSettings>
<add key="ClientValidationEnabled" value="true" />
<add key="UnobtrusiveJavaScriptEnabled" value="true" />
<add key="ClientId" value="aaaaaaaa-beed-face-bbbb-cchcchcchcch" />
<add key="ClientSigningCertificatePath" value="C:\\Certs\\myCert.pfx" />
<add key="ClientSigningCertificatePassword" value="p@ssw0rd" />
<add key="IssuerId" value="bbbbbbbb-beef-face-bbbb-cchcchcchcch" />
</appSettings>

The main difference here is that you don’t just use a ClientId/ClientSecret - Combination for an OnPrem-Installation but have to use a Certificate. I will not describe here how to obtain it. It is just the “normal” OnPrem-procedure for provider hosted apps. Also be aware that the IssuerId is not the ClientId.

If you need help on this part: There are plenty of documents about this. For example onMSDN

Backend, Webjobs and other projects that talk directly to SharePoint

App.Config

If you have any other Projects than the Web Project and the AddIn, you might have to do some extra steps. For example if you use a backend-project, you have to modify that, too. Ignore this chapter if you don’t have any additional projects.

But if you have any console application (like Azure WebJobs) you have to modify those configs, too. Sadly. Config-transform does not work out of the box with App.Config - files. But there is a way to change that.

First: Create your Config Files manually. So select Add - New Item and select “Application Config File”. Name it App.OnPrem.config. Now open the csProj-File with Notepad. Look for an ItemGroup containing the App.Config. There already should be an App.Debug.config and App.Release.config with a “DependentUpon” - Attribute. Just copy that attribute to your newly created files. So that you end with something like this:

<ItemGroup>
<None Include="App.config" />
<None Include="App.Debug.config">
<DependentUpon>App.config</DependentUpon>
</None>
<None Include="App.Debug.OnPrem.config" >
<DependentUpon>App.config</DependentUpon>
</None>
<None Include="App.Release.OnPrem.config" >
<DependentUpon>App.config</DependentUpon>
</None>
<None Include="App.Release.config">
<DependentUpon>App.config</DependentUpon>
</None>
</ItemGroup>

After that all your OnPrem-Configs should be displayed as children from App.Config in Visual Studio (2015). Now transform should work. You can check your csproj - File. If there is something like <UsingTask TaskName=“TransformXml” ../> in it you are ready to go. If not, you are probably using an older Visual Studio version. I would not advise it, but you can go with it, but have to do some extra steps: XDT Transforms for older VS Versions

Now its the same fun again with the config files as before. So add the same stuff into the App.*.config - files as you have done with the web.configs. In addtion add the transform Property to the configuration tag, so it looks like this:

Test what you have done until now

Now you should build your Solution for all four configurations. Check if all DLLs where created correctly for the O365 and OnPrem-Configurations. Deploy your application still with the O365 - Settings. Everything should look like before.

Changes on the Web Project

Now it is time to modify your main SharePoint-Project.

First of all: Add your SharePoint-Project to the build configurations. The good thing is: This time you don’t have to care about the configs. There will most likely be an app.config in your SharePoint-project, but you can just ignore it, because there are no real references in this project.

SharePoint - URL

The (nearly) final step is to put the right SharePoint-URL to your AddIn-Project depending on your Configuration. The simple way is to just enter the right URL each time you start debugging. The more complex (but better) way is to go the same way as you did with the MVC (Server) part. To do this, enter your Build-Configuration Settings again and set “Debug.OnPrem” and “Release.OnPrem” for your AddIn-Project.

The URL is stored in the user-settings instead of the project settings. But you can use the same magic to make a conditional path there. Open the file YourApp.csproj.user and add the condition to the “SharePointSiteUrl” - entry:

<PropertyGroup>
<SharePointSiteUrl Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">http://myDevMachine:10000/sites/Dev\_Ole/</SharePointSiteUrl>
<SharePointSiteUrl Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">https://my365AppName.sharepoint.com/sites/dev-ole/</SharePointSiteUrl>
</PropertyGroup>

IF you want you can also simply delete the user-file and put these values into your main .csproj. Sadly this cannot be loaded “on the fly” by switching your Build configuration . You have to unload and reload your project for this setting to be re-read. This is no issue when deploying your solution using an automated build (like tfs online), but while debugging locally.

Target office version & SharePoint-URL

The csproj - file contains an entry called “TargetOfficeVersion” which should contain the value “16.1”. Just like before we need to modify this using conditional values.

<TargetOfficeVersion Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">16.1</TargetOfficeVersion>
<TargetOfficeVersion Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">15.0</TargetOfficeVersion>

You can validate this setting while selecting the properties of the project and display the “Sharepoint”-Tab. Under “Target Sharepoint version” it should say “SharePoint 2013” if 15.0 is used or “SharePoint Online” if 16.1 is used.

Authentication

Office365 uses OAuth for Authentication. For SharePoint 2013 you have to use Windows Authentication instead. As I wrote before: There are a few documents out there explaining the Certificate-Part. But you also have to tell your project to use Windows Authentication. Again we use the same conditions like before, but this time use it for the IISExpressAuth:

<IISExpressWindowsAuthentication Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">enabled</IISExpressWindowsAuthentication>
<IISExpressWindowsAuthentication Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">disabled</IISExpressWindowsAuthentication>

AppManifest

The Appmanifest does not allow any conditional entries. That’s why we have to create a copy of it and do the conditional stuff inside the csproj file. So first create a copy called AppManifest.OnPrem.xmlns

In that file change the “SharePointMinVersion” from 16.0.0.0 to 15.0.0.0 in the “App"-Section.

Now it is finally time again to modify the csproj - File. You should be used to it by now :)

<ItemGroup Condition="'$(Configuration)'=='Debug.OnPrem' Or '$(Configuration)'=='Release.OnPrem'">
<AppManifestFile Include="AppManifest.OnPrem.xml">
<SubType>Designer</SubType>
</AppManifestFile>
</ItemGroup>
<ItemGroup Condition="'$(Configuration)'=='Debug' Or '$(Configuration)'=='Release'">
<AppManifestFile Include="AppManifest.xml">
<SubType>Designer</SubType>
</AppManifestFile>
</ItemGroup>

Things to mention on V 15.0 (SharePoint 2013)

While most of the code works on Version 15.0 as well on 16.1, there are some minor differences between these two. The following issues appeared on my project: - FieldUserValue has no Property “Email” - Microsoft.Client.SharePoint.Folder has no “Exists()” Both work on 16.1 but not on 15.0. So you have to write your own code to get around this issue.

Custom Compiler Symbols

Because of the differences between V16 and V15 you might want to have custom code depending on your configuration. Just like the DEBUG - Symbol you can add your custom Symbol for the OnPrem-Configurations. Just select the properties of each project you need that symbol in, select the Build - tab and into “_Conditional compilation symbol_s” enter “ONPREM” (or whatever you like):

Afterwards you can use the following code:

#if OnPrem
var userMail=GetMailFromUser(fieldUserValue); 
#else 
var userMail=FieldUserValue.Email;
#endif

Final thoughts

You are done now. Just keep in mind: If you use any Azure-connections (e.g. for caching, insights, blob-storage) you will have to handle that, too if you need to get rid of any external connection. The custom Compiler Symbols should help you with that task

Wayback Machine and 301 redirects

dotnet@oles.cloud (Ole Albers) — Tue, 15 Nov 2016 12:00:00 +0100

As you might know: the wayback machine from archive.org saves snapshots of websites. It does this automatically or can be forced to do so. However. If you ever had a 301 redirect on your site you are very much doomed.

Like browsers also the wayback-machine caches the redirect and does not continue to take snapshots from your sites. Even if you enter your url into the wayback-machine, you still will be redirected.

You can however force it to ignore the cache by manually store your site again:

https://web.archive.org/save/[YOURSITE]

for example:

https://web.archive.org/save/http://blathering.de

Et voílà: Everything works nice from now on.

[Image is CC-BY-NC-ND © David Baldingerhttp://www.dbaldinger.com/opinion_cartoons/second_page/wayback.html]

Getting rid of Proxy limitations using Fiddler

dotnet@oles.cloud (Ole Albers) — Fri, 16 Sep 2016 12:00:00 +0200

If you work in a company that uses Proxy authentication (aka forces you to enter a password to access websites) and are a developer you know my pain: Some Most applications just don’t really like proxies. Visual Studio itself for example does not like proxy authentication at all and responses with a wide variety of error codes (407 for example).

You can get rid of all these errors through different workarounds, but these are a real PITA.

In addition the application you code itself needs Internet access, for example if you develop Office365 AddIns, you are very much screwed.

After a long research I finally found a solution using Fiddler.

Most developers already use Fiddler for debugging reasons so there isn’t even any software that needs to be installed.

First of all:

Start your computer. You should not have been entered your Proxy credentials yet. Use Fiddler as proxy in your IE and enter your real Proxy in Fiddler:

Default Proxy-Settings to use Fiddler

Proxy-Settings in Fiddler

Do not select “System Proxy”. Otherwise. Fiddler will enter itself here, because Fiddler is the System Proxy right now.

Now open your favorite browser (except FireFox, as it uses its own proxy) and open any site that opens up the Proxy authentication window. Enter your credentials and load that site. In Fiddler you should now see (at least) one entry with response code 407 followed by a few entries with 200:

Select the entry with StatusCode 200 and display the Request Headers. Under “Security there now is a line with “Proxy-Authorization: “…

Copy that line.

Now Enter Ctrl-R in Fiddler to open the rules file. Look for

static function OnBeforeRequest(oSession: Session) {

in that file and add the following line directly at the top of the function:

oSession.oRequest[“Proxy-Authorization”] = “Basic DBDDHKPadasdkahsdasd=”;

where you replace the token behind “Basic” with the value you copied before.

Now save CustomRules.js and from now on no application has to enter any more Proxy passwords (as long as you use Fiddler as System Proxy)

If you are forced to change your Password, remove this line in the configuration file, and repeat the procedure from the beginning with the new password.

That’s it, mostly. For more comfort I would recommend to use the proxy switcher if you have to change the proxy more often.

Displaying Data from HomeMatic Actors on a WebInterface

dotnet@oles.cloud (Ole Albers) — Mon, 11 Jul 2016 12:00:00 +0200

This tutorial assumes that you already installed and configured FHEM and know how to use AngularJS.

As you might know I use my Raspberry PI to automate tasks at home. I already detect my SmartPhone, and my FitBit and also control my Webcam and Harmony remote using a raspberry and a web interface.

Now I also wanted to include actors from HomeMatic thermostats into my configuration and display it in a nice way.

Hardware used:

HomeMatic Wireless Configuration Adapter LAN (HM-CFG-LAN)
HomeMatic Thermostate HM-CC-RT-DN
Raspberry Pi V2.

Software used:

HTML/CSS
JavaScript
AngularJS
FHEM
PHP (optional)

Before we begin make sure you have already installed FHEM and the hardware for it. I will not discuss that here as there are a lot of good tutorials for that. A good (german) tutorial can be found in the FHEM Wiki. (If you got any english documentation, feel free to post it in the comments.)

I wrote a “general” PHP script to send commands and receive commands from FHEM. This can also be done without PHP and pure JavaScript-Code. I chose this solution because my configuration website is accessable through the internet, but my FHEM isn’t. So JS-Could (running on the client) would not work, but PHP (running on the server) does.

The PHP-Script:

<?php
 $host="http://192.168.178.36:8083/fhem?";

 $sensor= $\_GET\['sensor'\];
 $param= $\_GET\['param'\];
 $action= $\_GET\['action'\];

 if ($action=="get") {
 $get="detail=SENSOR&dev.getSENSOR=SENSOR&cmd.getSENSOR=get&arg.getSENSOR=param&val.getSENSOR=PARAM&XHR=1&addLinks=1";
 } else {
 $get="cmd=set%20SENSOR%20desired-temp%20PARAM&XHR=1";
 }

 $replaced=str\_replace("SENSOR",$sensor,$get);
 $replaced=str\_replace("PARAM",$param,$replaced);
 $total=$host.$replaced;
 
 $ch = curl\_init();
 curl\_setopt($ch, CURLOPT\_URL, $total);
 curl\_setopt($ch, CURLOPT\_RETURNTRANSFER, 1);
 $return = curl\_exec($ch);

 echo $return;

?>

The only thing you need to modify here is the IP-Adress (or hostname) and maybe the port from the $host - variable. As you can see, this is a simple GET-Request which could easily used through JavaScript (or even just inside a href) also.

You can test this script by simply calling it with some parameters:

http://yourserver/fhem.php?action=get&sensor={sensorname}_Clima&param=measured-temp

Replace “Sensorname” by the name of your sensor. You should receive the current temperature of the sensor as a result.

Now it is time to display this data in a fancy way in html. Create any HTML - Page using AngularJS. To use these nice Gauges you need ng-Google-Chart

Now lets see the HTML-Part containing the AngularJS-Directives to display and set the thermostate values:

<html><body ng-app="piApp">
  <div class="container" ng-Controller="heatController" ng-init="getStats()">
        <div class="row">
            <div class="box">
                <div class="col-lg-12">
                    <hr>
                    <h2 class="intro-text text-center">Heizungs
                        <strong>Einstellungen</strong>
                    </h2>
                    <hr>
                </div>
                <div class="col-md-12">
                    <div ng-repeat="heater in heaterSettings.allHeaters" class="heater">
                        <div class="battery"><i class="fa fa-4x fa-{{heater.battery.icon}} {{heater.battery.color}}"></i> </div>
                        <div google-chart chart="heater.gauge"  ></div>
                        <div class="caption">{{heater.name}}<br/>
                            <select ng-model="heater.ui.wish" ng-change="setTemperature(heater.id,heater.ui.wish)">
                                <option ng-repeat="option in availableTemperatures()" value="{{option}}">{{option}}</option>
                            </select>
                        </div>
                    </div>
                </div>

        </div>
    </div>
</div></body></html>

Some explanation:

piApp is the AngularJS - App for the website, getStats() loads the current Sensors using FHEM.php mentioned above directly from the start. This html uses Bootstrap (for repsonsive design) and Font Awesome to display the Battery indicator. Both use CSS to achieve this. (The Gauge-Display works without these libraries)

The Angular-Controller looks like this:

angular.module('piApp', \['googlechart'\])
    .controller('heatController', function($scope, $http) {
        $scope.url='fhem.php?';
        $scope.heaterSettings={
            allHeaters:\[
                {name:'Wohnzimmer', id:'Wohnzimmer\_Clima', ui:{},battery:{},gauge:{}},
                {name:'Schlafzimmer',id:'Schlafzimmer\_Clima',ui:{},battery:{},gauge:{}},
                {name:'Bad',id:'Bad\_Clima',ui:{},battery:{},gauge:{}}
            \]
        };

        $scope.getParamUrl=function(sensor,param) {
            return "action=get&sensor="+sensor+"&param="+param;
        };
        $scope.setParamUrl=function(sensor,param) {
            return "action=set&sensor="+sensor+"&param="+param;
        };

        $scope.getBattery=function(sensor, target) {
            $http.get($scope.url+$scope.getParamUrl(sensor,'batteryLevel'))
                .success(function(data) {
                    $scope.setBattery(target,data);
                });
        };

        $scope.getStats=function() {
            $scope.heaterSettings.allHeaters.forEach(function (heater) {
                // gauge:
                heater.gauge={};
                heater.gauge.type="Gauge";
                heater.gauge.options = {
                    width: 250,
                    height: 250,
                    redFrom: 20,
                    redTo: 30,
                    greenFrom: 0,
                    greenTo: 10,
                    yellowFrom: 10,
                    yellowTo: 20,
                    minorTicks: 10,
                    majorTicks: \['0','5','10','15','20','25','30'\],
                    max:30,
                    animation:{
                        startup: true,
                        duration: 1000,
                        easing: 'inAndOut'
                    }
                };

                // Gemessene Temperatur:
                $http.get($scope.url+$scope.getParamUrl(heater.id,'measured-temp'))
                    .success(function(data) {
                        $scope.setData(heater.gauge,data);
                    });
                $http.get($scope.url+$scope.getParamUrl(heater.id,'desired-temp'))
                    .success(function(data) {
                        $scope.setWish(heater.ui,data);
                    });
                $scope.getBattery(heater.name,heater);
            });
        };

        $scope.setTemperature=function(sensor, temperature) {
            $http.get($scope.url+$scope.setParamUrl(sensor,temperature))
                .success(function(data) {
                    console.log("Temperatur von "+sensor+" auf "+temperature+" Grad eingestellt");
                    //$scope.getStats();
                })
                .error(function(data) {
                    alert("Fehlgeschlagen");
                }
            );
        };

        $scope.availableTemperatures=function() {
            var available=\[\];
            for(var i=5;i<=30;i=i+0.5)  {
                var iasString= i.toString();
                if (Math.round(i)===i) {
                    iasString+='.0'
                }
                available.push(iasString);
            }
            available.push("on");
            available.push("off");
            return available;
        };

        $scope.setData=function(target,data) {
            var value=parseFloat(data.replace(/(\\r\\n|\\n|\\r)/gm,""));
            target.data=  \[ \['Label', 'Value'\], \['', value\]\];
        };

        $scope.setWish=function(target,data) {
            var value=data.replace(/(\\r\\n|\\n|\\r)/gm,"").trim();
            target.wish=value;
        };
     
        $scope.setBattery=function(target,value) {
            var value=parseFloat(value.replace(/(\\r\\n|\\n|\\r)/gm,"").trim());
            var color='green';
            var icon='battery-full';
            if (value<2.1) {
                color="red";
                icon='battery-empty';
            } else if (value<2.3) {
                color="red";
                icon='battery-quarter';
            } else if (value<2.6) {
                color="yellow";
                icon='battery-half';
            }
            target.battery={
                value:value,
                color:color,
                icon:icon
            }
        };

    })

Explanation:

“googlechart” in the module-configuration allows to paint those fancy gauges.

in $scope.heaterSettings.allHeaters you have to modify the Names of your Sensors. Enter as many sensors as you want.

All other functions communicate with fhem.php or display values.

It's because they at Microsoft hate web developers

dotnet@oles.cloud (Ole Albers) — Tue, 14 Jun 2016 12:00:00 +0200

That remark isn’t mine, it is from StackOverflow. :)

I had an issue that my Office365-App did not work correctly on the machine of ONE user. Sadly though that user was the Microsoft validation crew. I was unable to reproduce the error and did not even see any clue in the Server logs. Everything seemed to work fine.

Finally I managed to reproduce the error. It only happened under the following conditions: 1. Internet Explorer 11; IE - what else :) 2. Caching in IE has to be on “Automatic” 3. Developer Console MUST NOT be open 3.1. In fact: Even if you open it and close it afterwards, the error does not appear anymore 4. I had to retract/reinstall the app each time to check this behaviour which took about 10 minutes every time

I had issues with console on older IEs before, but at least angular uses some magic, so that issue is resolved. Nearly all questions in the web combining IE and console repeat the solution that you may not use “console.log()”.

But the real issue was: Caching. IE - in its wisdom - decided that the API-Call that returns the installation state should be cached. That cached value always responded that the app is not installed, even if it was. So the user kept receiving the “please install” - Page.

Any other browser (Chrome, Firefox, …) doesn’t cache these calls. And the IE doesn’t either when you open the Developer Console (what you do if you want to debug an application) which makes it really hard to track that error.

So the solution is to make sure the call isn’t cached using No-Cache-Headers or just by adding a random value to the target address. And of course resist the urge to display a reminder to use another browser. That could be bad for the validation results :)

FlashAirDownloader 2

dotnet@oles.cloud (Ole Albers) — Sun, 01 May 2016 12:00:00 +0200

⚠️ Warning

Development stopped! Please read this before downloading FAD

📝 Note

Keep your Version updated! Always click onto the “Check for Updates” in the Program after installation

The FlashAirDownloader 2 is out there (Because of my Laziness I will just call it FAD from now on)

A lot of people used the FlashAirDownloader which was meant as a personal tool and quick hack to access my Toshiba FlashAir Card. As more and more people asked me for features (and some for the code) I decided to make it open source so more folks can help improve the tool.

Sadly so: The code was a mess. As I told you: Just a quick hack. It would be to embarrassing to make this public :D So I decided to rewrite it from scratch: This time making it right regarding code quality and UI design. Nonetheless this program still uses Framework 4.0, cause I know there are Windows XP users out there who have no other possibilities to access the FlashAir.

But enough of talk. Here is the new FAD:

You can download the source directly from Github. Feel free to make the code better. Or download the binary here:

(The download from the old site is gone, please build from source.)

You will get an annoying amount of warnings from your browser and/or operating system. This is because the file isn’t signed. Valid signing requires a valid certificate which does not come for free. (and requires some amount of work) Feel free to build from source in GitHub if you have any doubts.

First start

After Installation and starting FAD you should see something like this:

If your FlashAir - card is already configured it should be found and the display should change to this:

For now: Keep calm and do not click on any of the blue squares. Use “File/Settings” instead.

Setup

Program Settings

Check if your settings are ok. Here is a small description from the settings:

FlashAir Url This is the Url from your FlashAir. You can try if you can access your FlashAir by entering that address into your Browser
Local Path Here the Images will be saved when using the Auto-Mode
Existing Files (Auto mode) Decide what to do if the file already exists.
Delete files after downloading (Auto mode) If checked, the files will be deleted after a successful download from the FlashAir
Flashair Imagepath Where your Images are located (may contain subfolders)
Filetypes to copy (Auto mode) Decide if you only want image files or other files, too
Load Thumbnails If enabled Thumbnails from Images on the FlashAir will be collected and shown. Might take a while. Thumbnails for local files are always displayed regardless of this setting.
Automatic folder Creation (Auto Mode) * Multiple Cards Use your Card ID or AppID to create subfolders if you use more than one card * Create By Date Decide in what format the directories should be named * Custom You can als create your own custom path. {0} stands for the CardId, {1} for the AppId and {2} for the current date. Examples about the date-format can be found here
Styling Decide if (and what) images should be displayed on the start page and the settings page of FAD.

Don’t forget to click onto [Save] under Card Settings in the lower left area of the screen.

Card Settings

Now after the application is set up it is time to check your FlashAir-Card. Just put your SD-Card into the computer and click onto [Load] in the upper left corner. After a few seconds three new tabs should appear: “Vendor”, “WLANSD” and “Disable Functions”

Click onto “Vendor” to setup the most important settings. I will only describe the most critical values. A complete list is available at the Toshiba FlashAir Website.

First make sure the Connection Timeout is correct. It must be at least 1 Minute. You can also set it to “0” to disable timeout completely. “App-Info” and “Card ID” can be modified if you want to use those values in the automatic folder creation.

If your Network settings already work, don’t touch them here. Otherwise follow the instructions on the FlashAir website.

Now make sure the “Upload Directory” fits the “Flashair Imagepath” in the “Application Settings”.

The last setting to check is “Upload Enabled”. This is required if you want to be able to save files onto the Flashair or want to delete files after downloading.

Now all you have to do is to click [Save] in the upper Left region and FAD is ready to go.

For the first run I would advice just keeping your card in your card-reader. It works exactly there like in your camera, but you can modify your settings more easy this way.

Auto - Mode

After FAD found your card click onto “Auto Mode” to start automatic downloading of your data:

At first the files are loaded recursively from your card. In the next step (if activated) the thumbnails gets downloaded. Then the files are being downloaded and deleted (if selected).

If the thumbnails cannot be retrieved red squares are displayed instead.

File Explorer

The file explorer does not download automatically images from the Flashair. Instead you can browse through your card and copy files manually. Just mark them by clicking and select the [«] or [»] - Button to copy them#

Known issues

https://www.youtube.com/watch?v=PtXtIivRRKQ

The FlashAir is sometimes a bit unstable (at least mine). There are a lot of reasons why something goes wrong. That’s why FAD tries up to five times to download or upload data. (Just Thumbnails are only tried once). There are many reasons why a download does not work. There can be a timeout, the Card can “crash” (it is a webserver running on it) or the camera just goes into standby. So often the best solution is to try again and or restart the camera and re-enter the FlashAir.

Any more questions?

Just look at the documentation on GitHub or open an issue there if you encounter any problems.

Love it?

Feel free to send some money to my Panama-Account

Detecting Smartphone via Bluetooth on Linux (Raspberry Pi 2)

dotnet@oles.cloud (Ole Albers) — Sat, 27 Feb 2016 12:00:00 +0100

After detecting my phone using WiFi (which does only work under Marshmallow while the phone is loading or the display is enabled) and detecting the fitbit device I came back to my original task to detect my phone using Bluetooth.

While there are many documents out there that describe this on different Linux and even raspberry variants none of those really worked for me. The raspberry 2 (Debian Jessie) seem to be a bit different there.

First of all you need a Bluetooth USB dongle. I buyed the Logilink BT00015 (available at amazon.com, amazon.de or elsewhere). I would expect that others work, too. But as always: On Linux, drivers can be tricky and it’s always good to know working hardware.

Before installing drivers it is always a good idea to update everything:

sudo apt-get update
sudo apt-get upgrade

Then install the libraries needed for bluetooth

sudo apt-get install –no-install-recommends bluetooth

I received an error that some dependencies could not be resolved. I fixed that by entering the following commands:

sudo dpkg --configure -a
sudo apt-get install -f
`Now insert the USB dongle and restart the raspberry`
sudo shutdown -r now

To check if the USB-dongle is detected, enter the following command:

lsusb

This lists all USB devices. There should be one offering Bluetooth - services

Now lets check if the Bluetooth itself works:

/etc/init.d/bluetooth status The response should now contain the word “Running”:

pi@raspberrypi:~ $ /etc/init.d/bluetooth status
● bluetooth.service - Bluetooth service
 Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled)
 Active: active (running) since Sat 2016-02-27 19:31:45 UTC; 17min ago
 Docs: man:bluetoothd(8)
 Main PID: 7841 (bluetoothd)
 Status: "Running"
 CGroup: /system.slice/bluetooth.service
 └─7841 /usr/lib/bluetooth/bluetoothd
`Now it's time to check for devices. Do nothing with your phone right now and enter:`
hcitool scan
`This should display all Bluetooth-devices in Range. As you haven't done anything with your phone yet it should not be displayed there. The output should be something like this:`
Scanning ...
 7C:2F:BE:EF:FA:CE Ole
 00:04:BE:EF:FA:CE SHIELD
 B8:86:BE:EF:FA:CE Fette Glotze
 88:53:BE:EF:FA:CE TOMATO-PC
 48:44:BE:EF:FA:CE TVBluetooth

(I modified the MAC addresses. I am not absolutely sure if anyone could do evil things using the MACs, but I prefer not to find that out :) )

Now put your phone into discovery mode. On Android Marshmallow you just have to enter your Bluetooth-settings for this. Then re-scan your devices. There should be one more, now:

hcitool scan

7C:2F:BE:EF:FA:CE Ole
00:04:BE:EF:FA:CE SHIELD
B8:86:BE:EF:FA:CE Fette Glotze
88:53:BE:EF:FA:CE TOMATO-PC
48:44:BE:EF:FA:CE TVBluetooth

F8:95:C7:H0:0H:0H G4

Now just note your Mac-Address. We will not pair our device. That takes a few steps and additional packages and additional problems. Pairing is not needed once we know the mac-address.

Now we cam just check the presence by using the info command. Simply leave your Bluetooth-settings on the phone and enter sudo hcitool info F8:95:C7:H0:0H:0H Event though your phone isn’t in discovery-mode anymore you will get some status information like the device name:

Requesting information ...
 BD Address: F8:95:C7:H0:0H:0H
 Device Name: G4
 LMP Version: 4.1 (0x7) LMP Subversion: 0x6109
 Manufacturer: Broadcom Corporation (15)
`if you turn off Bluetooth on your phone and re-enter that command you will receive a different response:`
Requesting information ...
Can't create connection: Input/output error
`So that's it. Just use the good old grep in your bash-script to check whether the phone is there or not:`
if hcitool info F8:95:C7:H0:0H:0H | grep -q 'Device Name'; then
 // Do whatever you like if you found it
fi

Again: Have fun :)

Tracking fitbit presence under Linux (Raspberry Pi 2)

dotnet@oles.cloud (Ole Albers) — Sat, 27 Feb 2016 12:00:00 +0100

As I wrote before I track the presence of my smartphone using WiFi. Since Android Marshmellow this doesn’t work anymore, because when switching to idle mode, the WiFi is disconnected. The easiest way to get around this issue would be to use Bluetooth instead. And in fact: I already decided to go that way. But then my company gave me a nice present: A “Fitbit Charge HR”. While my company assured, that the HR is not for Human Resource and does not try to track if I am programming enough I decided to give this a try.

Being a nerd I am not that much into that “fitness”-area. But I like the device as a small watch and the vibration alarm is nice too. I will surely sync this with my Appointments lateron. Getting some information about how many steps it takes to get the beer out of the cellar or the chips out of the kitchen is a nice benefit. Not really useful. But at least nerdy :)

So I plan to wear this device for a while. So it should be a good idea to use it for tracking my presence. Maybe in combination with a phone-detection as a fall-back. First I tried to use a standard USB-dongle to detect it. But as this device works on Bluetooth Low Energy (BLE) it cannot be detected with normal Bluetooth-Dongles. So hcitool scan or even hcitool lescan did not receive any results. Luckily the Fitbit ships with an USB-dongle. And even better, there is a package called “galileo” which purpose is to sync the fitbit. Installation is easy:

sudo apt-get install python-usb python-requests
sudo pip install galileo

Now install the dongle and restart the raspberry.

If you enter the following command sudo galileo -v you should see something like

2016-02-27 16:28:21,875:INFO: 1 trackers discovered All you have to do now to check if the fitbit is in range automaticly is some bash-magic:

if galileo -v | grep -q '1 trackers'; then
  // Do whatever you like if you found it
fi

Control your Harmony Hub with a Raspberry Pi (Linux)

dotnet@oles.cloud (Ole Albers) — Fri, 15 Jan 2016 12:00:00 +0100

So my surveillance cam now only records when I am not at home. But what if I forget to turn off the TV? That should happen automaticly, too! But fear not: There is a solution - at least if you own a harmony hub. With the help of the “HarmonyHubControl” - tool (written in c++) this is done quite easily.

First of all you need to put the source onto the Raspberry. Don’t be afraid. No rocket science here. All needed tools should be already on your Linux distribution. The easiest way would be to clone the source via git: git clone git://git.code.sf.net/p/harmonyhubcontrol/code harmonyhubcontrol-code If you prefer you may - of course - also use the long way and download those files in a zip file.

Now after you have the source code on your Pi you have to compile it. Done with ease. Just a single

make and you created the file “HarmonyHubControl” in that directory. Now all you need to know is the IP-address from your hub. You usually can find it in your WiFi-Router under the name “HarmonyHub”

Let’s try if the connection works:

./HarmonyHubControl \[Login\] \[Password\] \[Ip\] Warning! Double-check your password. If you enter a wrong one, you can still stert the activities, but on the next login in your “MyHarmony”-application you will be thrown out for a while. Don’t ask why this works this way. Better ask Logitech instead.

The Login is your email-address. If username and password are right you should see the following result:

So let’s list all activities: ./HarmonyHubControl \[Login\] \[Password\] \[Ip\] list\_activies As you can see there is an activity with the id “-1”. This activity shuts down all devices. For testing let’s just start an activity: ./HarmonyHubControl \[Login\] \[Password\] \[Ip\] start\_activity 15231836 does start the FireTV in my case, ./HarmonyHubControl \[Login\] \[Password\] \[Ip\] start\_activity -1 does shut em all down.

If you want to, do some fancy stuff with that possibilities. Use your stereo as an alarm clock? Have fun! (your neighbours, too, btw) :)

Switching the LED of a surveillance cam on and off using Linux

dotnet@oles.cloud (Ole Albers) — Fri, 15 Jan 2016 12:00:00 +0100

I already wrote a script on my Raspberry that starts and stops the surveillance mode of my camera using the “Surveillance Station” on the Synology DiskStation. Now I wanted to extend that tool and light the LEDs of the cam when it is recording and turning them off if it isn’t.

This has two reasons: First: I do not want to use the cam for “secret recordings”, damaging the private spheres of my visitors. Second: I can easily see if the cam is working as expected. Now my cam has the option to activate the LEDs or deactivate from a web interface, but it is all-or-nothing. It does not depend if it is recording or not. Which isn’t a surprise as the camera does not trigger the recordings itself but this is done by the Surveillance Station.

I own (and use) the “Robocam 4”, sold by 7Links. But this hardware is quite common and used under different names, too.

Finding out the right commands is quite straight-forward: Open the webinterface and listen what happens there using the developer console in Chrome:

The “Request URL” and “Form Data” are the most interesting informations here. Before entering that stuff into the Linux-shell let’s test this with a more “comfortable” interface. The chrome extension “Postman” offers a nice way to play around with POST-Requests. The setting should be “x-www-form-urlencoded” here:

If everything works as desired it is time to put that command to Linux using shell commands. Especially easy if the come does not only accept POST-requests, but GET-requests, too.

http://192.168.0.88/admin/system.cgi?led=2

worked like a charm in my case.

The only thing that’s left is the authentifivcation. If you receive a classic login-screen like from a .htaccess password-protection it can be easily added to the url. So wget http://username:password@192.168.0.88/admin/system.cgi?led=2 turns on the LEDs wget http://username:password@192.168.0.88/admin/system.cgi?led=0 turns them off

(0=off, 1=on, 2=blinking)

Just add these lines to your existing script and the LEDs should be work just like needed :)

Automate your Synology Surveillancestation with some simple Linux - commands

dotnet@oles.cloud (Ole Albers) — Thu, 07 Jan 2016 12:00:00 +0100

I own a Synology diskstation which has the “surveillance station” on board which records content from any LAN (or WiFi)-enabled webcam and sends an alarm when someone is moving at my home. It is possible to enter time schedules when the camera should be enabled and when it shouldn’t. But that does not really work the way I wanted. When I come home earlier or even have a day off I have to disable the camera manually.

So I decided that I want to use my smartphone to trigger the surveillance mode. Like most of us I always carry the phone with me. So it should be easy: If the phone is at home, I am, too. So the camera should be off. The same way when I am not at home.

The solution for this was surprisingly simple: A small bash-script first checks if the phone is in the network. The Surveillance station cam gets enabled or disabled according to this one condition.

But first things first. First of all, you should go to your routers admin interface and make sure, your phone will always have the same IP in your local WiFi. This works different on any reouter. On TP-Link it is called “reserve Ip”:

Now all you need is a Linux machine. I took a fresh Raspberry for this goal. Any simple distribution should be fine. We won’t need any special libraries.

After you connected to your Raspberry (or other Linux) create the following shell script via vi (or any other tool you prefer):

user='pi'
pass='circle'
server='sauger:5000'
phone='192.168.0.7'

dt=$(date '+%d/%m/%Y %H:%M:%S');
cd $(dirname $0)
touch lgstate.txt
read oldValue < lgstate.txt
if ping -c 3 ${phone} | grep 'bytes from'; then
  if \[ $oldValue != 'isda' \]; then
    echo \[${dt}\] 'stop recording'
    echo 'isda' >lgstate.txt
    #login
    wget -q --keep-session-cookies --save-cookies cookies.txt -O- "http://${server}/webapi/auth.cgi?api=SYNO.API.Auth&method=Login&version=3&account=${user}&passwd=${pass}"
    #disable
    wget -q --load-cookies cookies.txt -O- "http://${server}/webapi/entry.cgi?api=SYNO.SurveillanceStation.Camera&method=Disable&version=3&cameraIds=1" >dev>null
    #logout
    wget -q --load-cookies cookies.txt -O- "http://${server}/webapi/auth.cgi?api=SYNO.API.Auth&method=Logout&version=3" >dev>null
  else
    echo \[${dt}\] 'still there...'
  fi
else
  if \[ $oldValue != 'iswech' \]; then
    echo \[${dt}\] 'start recording'
    echo 'iswech' >lgstate.txt
    #login
    wget -q --keep-session-cookies --save-cookies cookies.txt -O- "http://${server}/webapi/auth.cgi?api=SYNO.API.Auth&method=Login&version=3&account=${user}&passwd=${pass}"
    #enable
    wget -q --load-cookies cookies.txt -O- "http://${server}/webapi/entry.cgi?api=SYNO.SurveillanceStation.Camera&method=Enable&version=3&cameraIds=1" >dev>null
    #logout
    wget -q --load-cookies cookies.txt -O- "http://${server}/webapi/auth.cgi?api=SYNO.API.Auth&method=Logout&version=3" >dev>null
  else
    echo \[${dt}\] 'still noone at home'
  fi
fi

Change “user” and “pass” in the beginning of the script with your user credentials. You can create a new user for this or use the “admin” - user. The user has to have admin rights sadly so don’t use this script on any machine that can be accessed by anyone but you.

“server” is your server address enter the hostname (or ip) and port there

“phone” is your phone’s IP-address

So what does this script do?

First it tries to ping your smartphone. If this is successful, it checks if this is the first time that works. If it isn’t the first time, nothing has to be done. But if the phone wasn’t accessable before it logs onto the synology and disables the camera using the web api.

The same way it enables the camera if the phone cannot be contacted anymore.

The script does only minimum outputs. If you run into problems, just remove the “>dev>null” part on any line to get more information.

Before you can start the script you should create an empty file where the state is saved by entereing touch lgstate.txt You can run that script by entering “./pingPhone.sh” on the command line. You can watch the changes live in the Surveillance Station -View of the Synology. Changes are displayed instantly. You do not need to refresh the page.

If you have more than one camera change the value of “cameraIds” to another value that is needed. You also can select multiple cameras by using commas: “&cameraIds=1,4,5”

Finally if you want to run this script automaticly you can create a simple cronjob for it

open crontab -e and enter the following line: \*/15 \* \* \* \* /home/pi/pingPhone.sh >>/home/pi/pingPhone.log (of course the path may differ on your system)

Now the presence will be checked every 15 minutes and the cameras will enabled or disabled accordingly. Now all you have to do is to modify your time schedule in the surveillance station that the camera is always in detection mode.

SCRUM and Git: Optimized Gitflow // with a lot of fancy colors :)

dotnet@oles.cloud (Ole Albers) — Wed, 04 Mar 2015 12:00:00 +0100

For a lot of teams, GIT is the first choice for maintaining the code. So it is for mine. But this post isn’t limited to GIT and should work with SVN or TFS with little modifications, too.

Why is the Git flow important when using SCRUM?

Aren’t those just two worlds, anyway?

Well. Kind of. But while SCRUM is quite flexible, scrums paradigm still has some very basic contents when it comes to manage projects:

You and your team work in sprints.
Inside these sprints there are user-stories you want to implement.

We decided to map sprints and user stories to our git flow.

Classic git flow:

You usually have the following structure in a git flow: (More fancy description and image at Atlassian)

You always got your master branch. From that branch you create single branches for each Hotfix. Your daily work is done in feature branches that start at the development branch. When a feature is finished, you merge it into your development branch. For a release, the development branch (which contains the added features) is the root of a new Release-branch. That release is tested and maybe modified until it fits the needs of your customer. When you are happy with that release you merge it back into the development branch as well as the master branch.

“Scrumified” Git Flow:

When trying to put SCRUM into the flow you keep your basic structure but add two additional layers to it. The single features are not created directly from the development branch. Instead at first for each sprint a new branch is created from the development branch. From the sprint there are branches created from each user story. After you’ve done that, the features are created from the corresponding stories. Merging back goes the same way. You merge your features back into the user story and the user story back into the sprint before merging the sprint back into your development branch.

This is a bit more work (and might look a little bit chaotic on these pictures at first) but it offers the advantages you have from “normal” git flow also to your sprint. Like before you only merge features that are finished. But now you also only merge user stories that are complete. And as a small bonus you can always analyze which user story was started and finished in which sprint. A Win-Win-situation like those unlikable marketing guys would call it. (If they were able to understand a single word I just wrote here) :)

IMAP-HTML-Autoresponder with PHP

dotnet@oles.cloud (Ole Albers) — Fri, 13 Feb 2015 12:00:00 +0100

Everyone knows spam. Everyone hates spam. And everyone who has his email-address written on the net (for example in the NIC-records for a domain) gets a lot of spam.

I am one of those guys who gets a lot of spam. Of course there are anti-spam solutions out there, but the best would be to not even receive spam. And spam filters need a lot of work and / or do not filter 100% perfect. That is why I decided to use a public email that has to go through a “gatekeeper”. A PHP-script that tells everyone that this is not my real mail address, but that there is another one where I could be contacted.

That email-address is not displayed in plain text but as an image to get rid of all those bots. SOME bots however should be able to get through: For example my hoster who needs to validate my email periodically. So these messages just are being forwarded.

I wanted to be as simple as possible. That’s why this script does not need much more than plain php. Only a replacement for phps “mail” was needed.

First step: Download swiftmailer. Just put the contents of the “lib” - directory onto your web space. That’s it. Ready to go.

Simple include the following line to access swiftmailer:

require\_once './swift/lib/swift\_required.php'; To access your mail you need to enter the IMAP-Settings. POP would work, too, but IMAP would be much more useful because we do not want to download the email-body of the messages processed:

// IMAP-Settings:
 $hostname = "{imap.hostname.com:993/imap/ssl}INBOX";
 $username = "username";
 $password = "password";
`Now enter your email-addresses. $targetMail is your real ("hidden") email-address where valid emails should be forwarded to. $thisMail is the public address that receives the spam.`
$targetMail="private@mydomain.com"; 
$thisMail="webmaster@mydomain.com";

Next you need to create an image with your favourite picture editing tool (GiMP, Paint.NET, PaintShopPro, etc.) This image should only contain your private email. Take a font you like. You might also use a “Captcha”-font. I - personally - hate that… :)

Upload that image to your web server and enter the url of that image. PNG would be a good choice.

$image=“http://www.example.com/images/mail.png";

Next, enter the text the sender should receive. By default everyone receives $templateHtml. For those who do not have the capabilities to display html (or don’t want to) a simple text-mail ($templatePlain) will be displayed. Users with a html-browser will see your email-image directly in the mail. Plain-text mail-clienst will display a link to the image instead.

$templateForward defines the mail content YOU will see if a mail is forwarded to you from a valid sender

So it is time to be creative :)

$templateHtml="<html><body>Thanks for your mail. Please note that this mail is no more. It has ceased to be! It is expired and gone to meet its maker! This is an Ex-Mail!<br/>But don't be afraid, there is a replacement for that. Just send your mail to: <br/><img src=\\"\_\_EMBED\_\_\\"/><br/><br/>If you cannot see that image: Click onto the following link to see my: <a href=\\"\_\_URL\_\_\\">new mail-address</a><br/>And no: This mail will not be forwarded. No one will see this until you forward it to my new mail-address. Thanks!</body></html>";

$templatePlain="Thanks for your mail. Please note that this mail is no more. It has ceased to be! It is expired and gone to meet its maker! This is an Ex-Mail!\\r\\nBut don't be afraid, there is a replacement for that. Just click onto the following link to see my: new mail-address: \_\_URL\_\_";

$templateForward="new message redirected";

As you can see there are two special tags in those templates. “__EMBED__” will be replaced with the embedded image (html-template only) and “__URL__” will contain a link to that image.

The last variable you can define is the white-list. Everything entered there is interpreted as a valid sender. Enter multiple values seperated by commas. $whitelist = "uberspace,denic.de,domainbox.net,romrobot.com,inwx.de,millerntor.hamburg,dotnet.work"; that’s all for configuration. Now just some explanation what the script does:

First of all the script checks your IMAP-Server for unread messages. It collects the header of each of those messages and compares the sender with the white-list. If the sender is recognized, the mail body is downloaded and sent as an attachment to the mail address you defined as $targetMail.

If the sender is not in the white-list, an automatic reply is generated using your html-template. It contains the image you downloaded as an attachment and embedded image of the mail. The last step is to mark the mail as read so it won’t be processed again.

In a perfect world…

Everything works now. But you can (and will) get problems sending the mail to the sender if the sender’s address does not comply with RFC 2822. While most spammers DO comply, mom DOESN’T… :) That’s because there is a mail client from a tiny little software company in Redmond, where they do not know much about standards. But this is no big problem at all. Just open lib\classes\swift\Mime\Headers\MailboxHeader.php from swiftmailer and put /* */ around the throw-part of the last function:

private function \_assertValidAddress($address)
{
   if (!preg\_match('/^'.$this->getGrammar()->getDefinition('addr-spec').'$/D', $address)) {
     /\* throw new Swift\_RfcComplianceException('Address in mailbox given \['.$address.'\] does not comply with RFC 2822, 3.6.2.');\*/
   }
}
`So finally here is the complete script.`
<?php
   require\_once './swift/lib/swift\_required.php';// IMAP-Settings:
   $hostname = "{imap.example.com:993/imap/ssl}INBOX";
   $username = "username";
   $password = "password"; // Script-Settings:
   $targetMail="private@example.com"; // "hidden" target mail-address
   $thisMail="webmaster@example.com"; // "official" mail-address (which receives the spam) 
   $image="http://www.dotnet.work/reply/mail.png"; // Image to be shown
   $templateHtml="<html><body>Thanks for your mail. Please note that this mail is no more. It has ceased to be! It is expired and gone to meet its maker! This is an Ex-Mail!<br/>But don't be afraid, there is a replacement for that. Just send your mail to: <br/><img src=\\"\_\_EMBED\_\_\\"/><br/><br/>If you cannot see that image: Click onto the following link to see my: <a href=\\"\_\_URL\_\_\\">new mail-address</a><br/>And no: This mail will not be forwarded. No one will see this until you forward it to my new mail-address. Thanks!</body></html>";
   $templatePlain="Thanks for your mail. Please note that this mail is no more. It has ceased to be! It is expired and gone to meet its maker! This is an Ex-Mail!\\r\\nBut don't be afraid, there is a replacement for that. Just click onto the following link to see my: new mail-address: \_\_URL\_\_";
   $templateForward="new message redirected";

   $whitelist = "uberspace,denic.de,domainbox.net,romrobot.com,inwx.de,millerntor.hamburg,dotnet.work";
 
   $allowedSenders=explode(",",$whitelist);
   $inbox = imap\_open($hostname,$username,$password) or die("Sorry, connection failed");
   $emails = imap\_search($inbox,"UNSEEN");
   $count=0; 
   echo "<pre>"; 
   if($emails) {
     $transport = Swift\_MailTransport::newInstance();
     foreach($emails as $email) {
        $count++;
        $header = imap\_fetch\_overview($inbox,$email,0); 
        $subject=$header\[0\]->subject;
        $sender=$header\[0\]->from;

        $messageHeader =  imap\_fetchheader($inbox, 1)
        if (strpos($messageHeader,"X-RESPONDERDATE")!==false) {
          continue; // no wild bouncing, please
        }

        if (strpos($sender,$targetMail!==false || strpos($sender,$thisMail!==false) {
           // We want no loops... :)
           continue;
        }
        echo "retrieving ".$subject." from ".$sender."\\r\\n";
        // Whitelist: 
        $isWhiteListed=false;
      
        foreach($allowedSenders as $allowedSender) {
           if (stripos($sender,$allowedSender)!==false) {
              $isWhiteListed=true;
              break;
           }
        }
 
        if ($isWhiteListed) {
           $email\_body = imap\_body($inbox, $email);
           $mailer = Swift\_Mailer::newInstance($transport);
           $message = Swift\_Message::newInstance()
              ->setSubject("Fw:".$subject)
              ->setFrom($sender)
              ->setTo($targetMail)
              ->setBody($templateForward)
           ;
 
           $attachment = Swift\_Attachment::newInstance($email\_body, 'origin.eml', 'message/rfc822'); 
           $message->attach($attachment);
 
           $result = $mailer->send($message);
           echo "whitelist redirect\\r\\n";
        } else { 
           $mailer = Swift\_Mailer::newInstance($transport);
           $message = Swift\_Message::newInstance();
           $swiftheaders = $message->getHeaders();
           $swiftheaders->addTextHeader('X-RESPONDERDATE', date('Y-m-d H:i:s'));
           $cid = $message->embed(Swift\_Image::fromPath($image));
           $html=str\_replace("\_\_URL\_\_",$image,$templateHtml);
           $html=str\_replace("\_\_EMBED\_\_",$cid,$html);
           $plain=str\_replace("\_\_URL\_\_",$image,$templatePlain);
           $message
              ->setSubject("Re:".$subject)
              ->setFrom($thisMail)
              ->setTo($sender) 
              ->setBody($html, 'text/html')
              ->addPart($plain, 'text/plain')
           ;
           $result = $mailer->send($message);
           echo "answered\\r\\n";
        } 
        imap\_setflag\_full($inbox, imap\_uid($inbox,$email), "\\\\Seen \\\\Flagged", ST\_UID);
     } 
  }
  imap\_close($inbox);
  echo "done redirecting ".$count." mails\\r\\n";
  echo "</pre>";
?>
`Just save this program as "reply.php", send test-mails and open "reply.php" in your browser to test the settings. When everything works fine you can simply add a cronjob so this script is run automaticly. The following example runs the script every 5 minutes:`
\*/5 \* \* \* \* php /var/www/reply.php

Just use the comments if you need any help

FlashAir Downloader for Windows

dotnet@oles.cloud (Ole Albers) — Thu, 25 Sep 2014 12:00:00 +0200

⚠️ Warning

Development stopped! Please read this before downloading FAD

📝 Note

**New Version available!**This version is obsolete. I would strongly suggest to use the FAD2 instead. You can download it here: http://dotnet.work/fad2.

Toshibas FlashAir is an SD-Card with built-in WiFi. It allows you to download Pictures from your DigiCam through WiFi. Sadly there is no working Windows-Application.

The FlashAir-Downloader solves that issue. It allowes Bulk-Downloading of your images. Your pictures can be saved in folders created automaticly containing date-information.

Installation:

Unpack the Zip and start the Setup.Exe

Start the Application and go to Settings:

Adjust the Settings if needed. Especially for “local Path”.

To use the Delete-Option you have to adjust the configuration of the SD-Card. Insert the SD-Card into your computer and open the file “CONFIG” file in the hidden “SD_WLAN”-folder with a text editor. Add the line UPLOAD=1 to the CONFIG-File.

Thats it. Close that window, turn on your camera and click onto “CONNECT”. If the connection works, you should see the folders and images of your SD-Card. Enter a folder or preview an image by double-clicking onto the icon. Mark the images you want to download and start downloading by clicking onto “Download”.

I give no guarantee about anything. This application shouldn’t do anything bad. But if your Camera explodes: Don’t blame me.This tool is and always will be free. But I don’t mind donations, of course :)

— Update V 1.1 —

I just updated the FlashAirDownloader to Version 1.1.

The following has changed:

Bugfix: If the file date is quite old (<2000) it could not be downloaded. This has been fixed. [*Yeah! Year 2000 bug in 2014. Well done, Toshiba *🙄 😁 ]
New Feature: You can now change the contents of the CONFIG – File directly. Just insert the SD-Card into your computer and Select “Card Properties
New Feature: If you use more than just One FlashAir-Card you can now automatically create subfolders for each card. You can use the internal “CID” ID for that or the Application ID which may contain any custom name.

(Tested with Windows 7 and Windows 8.1)

A new Version is available: http://dotnet.work/fad2

Emptying SharePoint-List the (quite) fast way

dotnet@oles.cloud (Ole Albers) — Fri, 10 Aug 2012 12:00:00 +0200

I already offered a solution to empty a SharePoint list at light speed

But this solution hast two big disadvantages:

You cannot empty a list that has more entries thatn the allowed list view threshold (default is 5000)
All Lookups will be lost

This solution has non of those drawbacks. It is much slower than that solution but also much faster than “normal” deletion of SPListItems or using a Caml-Query to do that.

Just be asure that this is not thread-safe. So make sure noone is writing into that list in the same time or that lines might not be deleted.

private int rest;
public static void TruncateList(Guid listId, int limit)
{
SPSecurity.RunWithElevatedPrivileges(delegate()
{
using (SPSite site = new SPSite(SPContext.Current.Site.ID))
{
using (SPWeb web = SPContext.Current.Web)
{
try
{
\_rest = limit;
web.AllowUnsafeUpdates = true;
SPList listCopy = web.Lists\[listId\];

ContentIterator iterator = new ContentIterator();
iterator.ProcessListItems(listCopy, ProcessItem, ProcessError);

web.Update();
web.AllowUnsafeUpdates = false;
}
catch (Exception ex)
{
if (!ex.Message.Contains("Limit reached"))
{
Debug.WriteLine(ex);
}

}
}
}
}
);
}

public static bool ProcessError(SPListItem item, Exception ex)
{
if (!ex.Message.Contains("Limit reached"))
{
Debug.WriteLine(ex);
}

return true;
}

public static void ProcessItem(SPListItem item)
{
if (rest<=0)
{
Debug.WriteLine("Limit reached");
throw new Exception("Limit reached");
return;
}
item.Delete();
\_rest–;

}

Emptying SharePoint lists the ultra fast way

dotnet@oles.cloud (Ole Albers) — Thu, 09 Aug 2012 12:00:00 +0200

SharePoint is damn slow when deleting lines from lists. Each entry can take up to some seconds which isn’t nice when you have to delete thousands of lines.

Also no Trunc-Methode is available.

Other solutions found on the net creating xml-files to bulk delete aren’t very fast either.

That’s why I decided to write my own fast truncate-method for SharePoint-Lists.

All content of that list will be deleted. Use at your own risk!

What it does:

First a template of that list is being created. After that the list is deleted completely. Than a new list using that template is being created and at the end that template is being deleted.

Works damn fast (just a few seconds even on big lists). IF anything goes wrong you can use the template to create a list manually.

public void TruncateList(Guid listID)
{
SPSecurity.RunWithElevatedPrivileges(delegate()
{
using (SPSite site = new SPSite(SPContext.Current.Site.ID))
{
using (SPWeb web = SPContext.Current.Web)
{
try
{
web.AllowUnsafeUpdates = true;
// alte Werte merken
SPList listCopy = web.Lists\[listID\];
string title = listCopy.Title;
string description = listCopy.Description;

string backupName = string.Format("BACKUP{0}{1:d} {1:HH}{1:mm}", title,DateTime.Now);

// save template
listCopy.SaveAsTemplate(backupName, backupName, string.Empty, false);
SPListTemplate template = web.Site.GetCustomListTemplates(web)\[backupName\];

// delete list
listCopy.Delete();
web.Lists.Add(title, description, template);

//  remove template
SPList gallery = web.Lists\["List Template Gallery"\];
foreach (SPListItem item in gallery.Items.Cast().Where(item => item.Title == backupName))
{
item.Delete();
break;
}

web.Update();
web.AllowUnsafeUpdates = false;
}
catch (Exception ex)
{
\_log.Error(ex);
throw;
}
}
}
}
);
}