Coincidentally, I've spent countless hours during the Covid-19 isolation looking for good to excellent textbooks on Engineering and Computer Science. Perhaps I can share them in the near future. These are some excellent books based on my findings for your particular requirements.

1) For general applied and engineering Maths book perhaps you can look into both textbooks by Prof. Kuldeep Singh and they uses Matlab but Octave should be fine [1].

a) Engineering Mathematics Through Applications b) Linear Algebra

2) For signal processing especially for communication I'd highly recommend "Digital Signal Processing in Modern Communication Systems" by Andreas Schwarzinger. It introduces the modern communication systems from the first principles to the more applied OFDM systems with examples that can be run using either Matlab or Octave. Just get the book it's really good even if you're not interested in communication systems.

3) If you want to learn more advanced applied engineering books on signal processing I'd recommend reading books on time-frequency techniques both linear and non-linear. For introduction check the book "Explorations in Time-Frequency Analysis" by Patrick Flandrin. Time-frequency analysis is most probably going to be pervasive very soon for increasing the accuracy of AI and ML by providing better intermediate processing of data features.

[1] https://maths-for-all.co.uk/