Is it hard? Depends on what you want to do. Since you have some flight sim experience, you won't find it hard to just fly around in the atmosphere. Anything beyond atmospheric flight is obviously going to be more difficult.
Once you get outside of an atmosphere, there are a whole new set of rules to learn, and that will take some time. Many of these rules are going to be counter-intuitive. Things like "Faster = Higher, but Higher = Slower" take some getting used to. Going faster means you have more centrifugal force, and that will cause you to increase altitude - but the farther you are from the gravity source, the less centrifugal force you need to maintain altitude, etc.
Go Play In Space is a great way to start, and I also recommend starting with a scenario that has you already in orbit - say the DG docked to the ISS (also, start with the Delta Glider, not the Atlantis). Experiment with Prograde (forward) and Retrograde (backwards) burns - use the Autopilots to maintain the right attitude for those burns. Watch the way the burns change your velocity, and the way velocity changes your altitude - and the way that changes in altitude change your velocity.
Earthbound transportation - even conventional flight - is largely two dimensional. While airplanes change altitude, the targets (airports) don't, and they don't usually move, either.
Spaceflight is four dimensional - the fourth dimension being time. Most of the targets (space stations, planets, moons, etc) are moving. You need to get to the right place, at the right time, AND be moving in the right direction when you get there! Fortunately, there are ways to do this that aren't as difficult as it sounds. It can be broken down into individual steps that aren't that difficult to achieve - once you learn to use the MFD's.
You don't need much for math skills to be good at orbiter - but you will need to learn to understand the basic concepts of Newtonian physics and Kepplerian Orbits. After a while, and some practise, this will come to you, and you will develop an intuitive understanding of Orbital Mechanics.
When you need to ask questions here, it helps if you know the "language". There are some terms you will need to know, like Periapsis and Apoapsis (and the abreviations like PeA, or Periapsis Altitude). Also, everything is relative, so always try to keep in mind what the reference is. For instance, if you are in LEO (Low Earth Orbit) and have a velocity of 7.515 Km/s, you should understand that that velocity is relative to a certain reference - in this case the center of the Earth (and that reference doesn't rotate with the Earth).
Yes, this sounds confusing. Have some patience, and it will come to you. I managed to figure it out, there's no reason you can't as well!